Use a single swap!. Place all operations on a map, including getting things, in that swap!. Otherwise, if you're accessing that map concurrently, you're asking for race conditions.

you don't need swap-if!, (swap! game-state #(if true % (....))) will be pretty much instant. It should just be comparing the revision ID of the atom

Is the main concern race conditions instead of performance?

(oh I guess unless you're using watchers ?)

@U02F0C62TC1 ok so my current method of doing the if check is fine, it will just check ids

No im not using watcher, the map contains a lot of entities designed to be used in ECS

If your main concern is performance and the most time is taken by those complicated transformer (probably is), then atoms are the least of your concerns, unless you're using and changing thousands of them.

Ok so i guess the overhead of swap is pretty low, at most i can change the system to do things in a single swap to avoid race conditions probably to avoid unwanted early cleanups

Thanks!

if some of the updates (excluding the ones who do no changes) are trivial (are they?) to calculate and some are expensive, isn't there a risk that those quick updates might starve the expensive ones?

you're right it would still bump the revision id :/

what's a revision id? don't atoms work over javas atomic references and do a compare-and-set with the old and new val in swap!?

I guess if i really want to I can just add a swap-if using compare-and-set

i was thinking of the "valueOffset" of the atomic reference

(i'm not 100% sure it gets bump though now)

yeah I may be 100% wrong on my understanding on how this work sorry, I'll try to test some stuff at some point

I don't understand what people are talking about. Swap takes care of race conditions by doing optimistic locking and retrying the update if needed.

For performance, you should be a little aware of contention. You still want to minimize how often swap has to retry, meaning if two threads enter the swap at the same time, one will have to retry, thus the time spent for the operation inside the swap will be doubled.

So I guess if doing multiple small swaps reduces contention, it would be better, otherwise a big swap is better.

As for your swap-if question... I guess to make that thread safe, you need to use compare-and-set! and do the retries yourself. You can't really do the if inside swap!, cause there's no way to cancel the swap really. Only way I can think is to use a validator, but that will throw, might be annoying. So probably you can implement your own swap-if!

(defn swap-when! [a pred f & args]
 (loop []
  (let [old (deref a)]
   (when (pred old)
    (let [new (f old)
          success? (compare-and-set! a old new)]
     (if success?
      new
      (recur)))))))

yeah i was basically planning to do this

instrumentation only checks :args , https://clojure.org/guides/spec#_instrumentation

> validating the implementation should occur at testing time I never understood this. And I know this was brought up and explained before. But still… Isn’t (clojure.spec.test.alpha/instrument) used exactly in the context where I’m testing? The word “test” is in the name of the namespace after all. I can’t tell the difference between these flavors of “testing times”. Can someone please explain the difference between these different testing contexts?

I'm no authority on the matter, but the way I think about it (and I may be wrong) is • instrument checks if the callers are behaving correctly wrt. the spec, either when running the app interactively or possibly when testing those other components • check checks that the function itself works according to the spec but perhaps someone on #C1B1BB2Q3 can give better answers

Thanks! That's unfortunate.

I had a memory that it's something like that. But that basically means that for actual runtime correctness checking and documentation with spec you need to use both {:post #(s/valid? ::input x)} and (s/fdef ... ).

there's also alternatives like https://github.com/jeaye/orchestra

Yes.

A tiny nitpick - x is not a var, it's just a binding. But not important in this case.

Clojure's locals clearing will be sure to null out the first x local, but even without that, JVM will see it's no longer used later in the method and will GC it.

This is how this code looks after being compiled and then decompiled back to Java. Observe x = null .

Funnily enough, it doesn't do much since there is final Object o = x before, so the reference is still seemingly held. But, again, JVM takes care of that, it will be seen as collectible by the GC as soon as it is used the final time.

@U06PNK4HG Is locals clearing useful on modern JVMs? I have no idea why it's there in the first place but perhaps older GCs needed it to release the memory held by x sooner.

AFAIU, locals clearing was introduced to accommodate laziness – so that as you iterate over a (possibly infinite) lazy collection, the head is not held by a local.

But I agree with you, I'm not sure it is needed at all right now. I need to see an example where it would achieve something that doesn't happen automatically.

But if JVM immediately sees that something is not used later on, why would the head be held?

Doesn't structural sharing play a role here? If there is a reference to the map and the key is still on some old copy of it (even if it's not accessible) then it's not expected to be cleared because it's not accessible by clojure convention and the JVM is probably not aware of it. If dissoc would be in-place it would change other refs. If dissoc re-constructed the map it would be very bad for performance. So I expect it's using structural sharing (I may be wrong though).

> But if JVM immediately sees that something is not used later on, why would the head be held? Yes, I can't come up with an example of how that is possible. But perhaps such example exists. Maybe, it has to involve closures, where JVM can't figure out a value is not used anymore across method boundaries.

> Doesn't structural sharing play a role here? If there is a reference to the map and the key is still on some old copy of it Sure, but that would be a different question then. If you have two references to the same thing, and dissoc only on one of them, that's quite obvious that the other would still hold everything in memory.

dissoc in this example only removes a key but a reference to the map still exists and the key isn't "physically" removed on every dissoc (since other copies may exist). You're suggesting the jvm "understands" it's the only ref because of the function call that generates it?

When you dissoc from a map, you create a new map with that key/value missing from it. The old map still "exists", but if nobody holds a reference to it anymore, garbage collector can now free it. > You're suggesting the jvm "understands" it's the only ref because of the function call that generates it It is the job of the GC to keep track whether each object in memory is reachable, that's its main job pretty much:).

JVM can't hold something in memory "just in case somebody uses it". It would never be able to reclaim such an object. It knows for a fact whether there are references to an object (hence it is kept) or there are none (hence it is freed).

I believe the problem was in code generation for closures where at one point they held references to more things than strictly necessary. I also believe this was fixed. That said, this is a vague memory and Alex could likely easily confirm or deny this as a potential cause.

the nice part about locals clearing is that you dont have to wait to exit the scope of a let form for dead objects within the lexical environment to get GC’d I think its still a useful optimization in 2023

The whole point is that being in a lexical scope of let does not prevent GC from happening. There isn't even such thing as "lexical scope of let" for the JVM, and even if it were, JVM still internally marks any local variable that is used for the last time in a method as collectible.

I assume that's because middleware is applied in order. So if you swap the order of those calls to restrict, the order of checks will probably change as well.

Right, but I cannot easily do that. The all routes restrict is in the top level namespace, the more specific rules are in the namespaces with specific routes.

It naturally follows that I would apply broad rules on the combined routes object rather than on each endpoint explicitly

That sounds like a general architecture problem then. If my first message was correct, then the combination of libraries that you use has very specific requirements, and you simply have to deal with that. Hard to suggest anything specific without reviewing the actual code, but in general depending on the loading order of namespaces should be avoided, precisely because it's harder to control than simple function calls. Things like Integrant or Component help here.

I think compojure is the culprit here, its wrap-routes function makes middleware apply in reverse as to what you would expect

If the class implements the interface, you call the static method on the class like normal: (SomeClass/someMethod args)

I you only have the interface, it's likely that the javadoc will point to one of the implementations (for example: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/io/InputStream.html has a list of direct known subclasses)

There is no class, that is the problem. I had never seen that before, but apparently it is a thing. Ironically the method I need to call is called builder and returns an instance of a class that implements the interface... no idea why things need to be that convoluted.

An example of calling a static method on an interface:

(java.lang.ProcessHandle/allProcesses)

https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html for those (like me) who don't know about "default methods" on interfaces

ah ... the builder pattern - it's used a lot in java land to make code "easier to read" ... usually the methods on the builder aren't static, but they're very often implemented as an inner class (which you can reference with a $ in the name https://clojure.org/reference/java_interop) ... you can use the clojure reflect api to tell you which classes in the hierarchy on the object (https://clojuredocs.org/clojure.reflect/reflect) and that'll tell you what methods are available from what declaring classes ... but yes interfaces can also have implementations now

I feel very stupid now... I just needed to call without importing... just like @U2FRKM4TW wrote... I am just so used to doing that on classes that it never occured to me to ignore the importing step... thanks all of you for the help!

Huh?.. Importing also works just fine - in fact, that's how I did that initially (or rather, Cursive did it for me).

(import java.lang.ProcessHandle)
=> java.lang.ProcessHandle
(ProcessHandle/allProcesses)
=> #object[java.util.stream.IntPipeline$1 0x6d041bab "java.util.stream.IntPipeline$1@6d041bab"]

Now I feel even more stupid... it turns out I typed the interface name wrong... it was KmsClient but I typed KMSClient... I probably owe you all a beer for wasting your time :rolling_on_the_floor_laughing:.

Here is one approach that doesn't use QosFilter. https://medium.com/helpshift-engineering/load-shedding-in-clojure-d4857ce11588

editor just let me jump to the code like to my code. I don’t use anything special.

I press cmd + down on ns or fn which I want to see in third party library

Same here, I just find the dependency's repo and make a local clone and then use my editor to browse it

as I'm making some exploration: first useful function is going to be loaded-libs in clojure.core, which lists the namespaces available to be required

additionally, clojure.repl/dir and ns-publics can be used to list functions provided by a namespace, and then clojure.repl/source can be used on the functions to view them

I just use editor 🙂

never complained

try Intellij + Cursive

depends on your preference

understandable, I'm trying to sus outs editor-agnostic methods (beyond evaluation); I'm quite happy with my Neovim/Conjure setup and have little intention to swap to anything else unless im paid to atm :^)

👍 just don’t go into the trap to make things harder, than they are ;)

with my years of keybind-accrual/muscle-memory, I think a swap to Intellij could be catastrophic for me ;^) im sure its nice, but nvim's been my home for a long time now

FWIW, there's IdeaVim that helps bridge the gap. It doesn't have all the features of course, but it's enough for my preferences. Any misalignment with a real Vim is outweighed by the features, in no small part by seamless code exploration where I can study even compiled Java code that doesn't have any sources. All at the click of a mouse button, with the ability to debug through that compiled code, search for usages across the whole classpath, etc.

(slurp ( "path-not-exist"))
Execution error (IllegalArgumentException) at price-sheet.main/eval70665 (form-init13554847912189378597.clj:1).
Cannot open <nil> as a Reader.

Thanks. But this is during the compile. And doing the same in main compiles fine. This is not during runtime.

then it means something call this fn during compiling while you don’t have the file. def ?

i checked for def and removed all of them. I renamed the method to junk and started over. Same error.

I’ll take another look for def usage.

it has to be somewhere

ok, are there any other cache locations? other than target/ folder?

I mean somewhere in the code

which is triggered during compiling instead of runtime

let me ask this question. If I have a single def in the file, but it is not used, will it still evaluate the whole file?

I think so

Oh ok. thank you. Let me try to remove it

Hey @U0WL6FA77 that didn’t work. 😞

I can paste the file here if you like?

sure

Do you have only 1 file for sure?

I’m not sure what you mean exactly. Are you saying “any” def any where in the source code?

I mean if you call load-yaml (directly or indirectly) from other ns

No I don’t. What’s also weird is that I put the exact same code in my “main” and that has lots of defs, and that works. 🤷

i try to keep most of the methods private

Clojure CLI version 1.11.1.1165

io.github.clojure/tools.build {:git/sha "e3e3532", :git/tag "v0.8.0"

is there a more recent build version?

The code above should be compiled just fine. If it fails during compilation, then something calls one of the functions during compilation. Or you have some rogue cache that somehow messes things up, so try removing the existing compiled classes. In any case, that exception should have a stacktrace. And that should help you figure out what exactly calls that failing function during compilation.

I don’t think this is version issue. It is like @U2FRKM4TW saying.

If you can’t find it, then comment “half of the code”, later on uncomment / comment other half of the code depends if error appear or not. You will narrow the scope.

sure, I’ll try that. I see the stacktrace, let me look

Ok, I think I see it. Its a chain of def’s referencing the config file.

thanks guys for your help. Let me refactor it and see

👍

Unless I'm mistaken, tools.deps (`deps.edn`) should pick up the newer version unless it was explicitly excluded or the older version was explicitly demanded in the top-level deps.edn. Would you be able to create a minimal reproducible example where it's not the case?

> And today we are getting failed builds without even updating our deps. Do you have any libraries in your dependency tree that have SNAPSHOT, RELEASE, or range versions? Including all the transitive dependencies.

We do have a snaphot dep, so that's probably the trigger. But the specific lib version wasn't changed in any of the snapshots.

To alleviate the snapshot-caused issues, you can depend on the timestamped version (or whatever it's called) - the exact version of a particular snapshot. But it won't solve your issue of some older deps for some reason trumping the newer ones.

> Unless I'm mistaken, tools.deps (`deps.edn`) should pick up the newer version unless it was Good to know. I suspect the issue is that we are still building with lein through a deps.edn plug in.

Oh. That's the first thing I'd blame, yeah.

> To alleviate the snapshot-caused issues, you can depend on the timestamped version Not sure this will help if the deps in the snapshot didn't change. Something about that dep tree algo shifted.

> Oh. That's the first thing I'd blame, yeah. Thanks.

I would not recommend using a lein plugin to consume a deps.edn

You are likely to get slightly different depending on which tool you use if you do

lein’s dep resolution uses (i believe) the first version it comes across. so you won’t get consistent resolution build to build. (please correct me if i’m wrong)

Last time I looked at a plugin for lein to get deps from a deps.edn file, what I found was a pretty shallow read the deps from deps.edn then pass those deps to lein's machinery to build out the rest of the dependency tree

lein uses mavens dependency resolution algorithm, it terms of resolving conflicts, where tools.deps has its own

Thanks alot all.

@U11BV7MTK more accurately, it uses aether, which chooses the dependency closest to the root of your dependency tree when there are conflicting versions of a dependency

deps.edn on the other just picks the latest version of everything and differs from maven/aether

though you can use :override-deps to change this behavior, and likewise managed dependencies for maven

I think that’s true for transitive dependencies. But uses versions which are “top level” in your deps edn file

This is how it should be. Gotta get the team over to deps.edn. Any dep that is top level should be #1.

Is there a lein pugin to build the tree the deps way lol?

why arent you using managed dependencies if youre using lein?

i think that is the #1 problem here, if anything — i always use managed dependencies in any monolithic project to ensure i have the exact same version of a library across all of them

likewise for deps.edn i always use override-deps to enforce consistent versions of libraries

Ooooh that sounds nice. I have not seen that.

https://codeberg.org/leiningen/leiningen/src/branch/stable/doc/MANAGED_DEPS.md

its existed for several years at this point, and in the java world, almost all maven projects use managed dependencies to do what i said (enforce consistent versions of all dependencies across projects)

🤯

Mind blown

This is my first distributed app project, so it hasn't been an issue until now.

i see; its also very convenient because you get to just update one entry in your project.clj and have it apply everywhere at the same time

likewise, for deps.edn, you can use :override-deps to achieve this effect

How does it work for :override-deps?

here is a simplified version of the clojure website’s example

{:deps {org.clojure/core.async {:mvn/version "0.3.465"}}
 :aliases {:old-async {:override-deps {org.clojure/core.async {:mvn/version "0.3.426"}}}}}

for leiningen, you would do something like

:managed-dependencies [[foobar "1.2.3"]]

Ohhh, I have seen this. Its not exactly the same issue as one lib bumping versions and the rest of the 30 (jeeze too many) apps loading in the newest version.

It solves some issues for sure. Our main libs are all snapshots, so I don't know if it works.

are they snapshots you control or snapshots of random 3rd party libraries

in general, the problem here is depending on snapshots: they are exactly how you have builds that work today and break tomorrow despite nothing changing

They are ours.

I can't put enough lols here. Yeah, I'm for versioning, but my boss likes snapshots.

pros / cons

well, it sounds like you guys are migrating over to deps.edn, so i recommend building a strong case for using :local/root when that is done, and preferably putting things in a monorepo so its easy to build and test everything at once

snapshots force you to go update everything.

alternatively, you can investigate tools like polylith, which work atop deps.edn and apparently make this very easy to manage

Come work for us, we agree.

A couple of us have looked at polylith. Its a good direction to go.

The root of this thread is that the maven dep tree algo sucks. deps.edn is a good path, and the lein deps.edn plugin erases the benifits.

hm, i personally find it okay — it works well for java, kotlin, and scala, and it seems to work fine in clojure, too. again, I recommend giving managed deps a try

you will very likely have to do this anyway when you migrate over to tools.deps and youre dealing with a bunch of internal projects that pull in random logging libs and depend on each other

deps.edn takes the latest, which I think should solve most of it. our libs are all non breaking.

managed deps is certainly a great tool we can use in some cases.

i mean, the core issue in the end is “we have conflicting versions of transitive dependencies, which one should we choose”? maven/aether decides “pick the one closest to the root of your dependency tree” and tools.deps decides “pick the latest no matter what” — if you want to enforce a version with maven, then you can put the library at the root of the tree or just use managed dependencies so you know exactly what version of everything you are getting

i dont think there is a clear answer to this question, both have their pros/cons

one reason i like maven approach a lot is because it helps ease onboarding java programmers (and other JVM language users) to clojure, what they expect across the JVM langs will work the same as in clojure

For sure

> pick the one closest to the root of your dependency tree Feels like it should pick the one in :dependencies but it doesn't always. What's the root of the dpes tree? I'll bet if you spent more time in java its an easy question. From clojure IDK...

well, when you run lein deps :tree or clojure -Stree for instance, “closer to the root” means you are in a lower indentation level of what gets printed

I hear ya

I have done that many times. clj / cljs. When its wrong, not sure how to change it.

Food for thought. My brain is full. I really do appreciate the convo!