It depends on the library. Look in the deps.edn file for the prep key and see what it "ensures" -- that's usually the folder (or file) that prepping will cause to be created, which is how the prep machinery knows whether it is needed or not.

See https://clojure.org/guides/deps_and_cli#prep_libs for an example.

The folders will be created in the checked out repo in ~/.gitlibs

Know if its safe for me to delete ~/.gitlibs ? Should get recreated right

It's not safe. It's like ~/.m2 -- if you delete it then any .cpcache folder entries already created by the CLI will point to none-existent files.

You'd need to use -Sforce on every CLI invocation that might reference a git dep, so that it would recompute the dependencies.

You can just remove the single folder that :deps/prep-lib specifies in :ensure from ~/.gitlibs/libs/<org>/<repo>/<sha>/deps.edn

(that's where you'll find the checked out repo -- so there will be a copy for every sha)

Cool, good to know

It is safe, .gitlibs is a cache. The CLI will notice the dir is missing and re acquire

You should not really need -Sforce

Oh... tools.deps hadn't used to notice missing artifacts in .cpcache tho', right? So that got fixed at some point?

yes, quite a while ago

well looking at the script again, it may be a little more subtle. if the gitlib dir itself is missing, that will be reobtained and if the prep dir is missing, that's the trigger for prep. other changes may not be noticed

missing .jar files (say in .m2) or stale manifest files in local or git all trigger the staleness check

Thanks. I ran into the "missing .jar" stuff a lot because I often blew away ~/.m2/repository while testing/trying to repro weird bugs to help people here on Slack 🙂 I haven't experimented too much with blowing away parts of ~/.gitlibs -- I have blown away the whole directory a couple of times (I just checked and I have over 7,000 deps.edn files under ~/.gitlibs/libs/ so it's probably time to blow it away again 🙂 )

there are also two parts to gitlibs - the git object dirs and the worktrees (effectively checkouts at a commit, which are added to the classpath and where builds happen)

Apparently, I have 21G of stuff in ~/.gitlibs/ 😐

Adding param-tags helps:

user=> (map ^[] String/.toUppercase ["foo" "bar"])
Syntax error (IllegalArgumentException) compiling String/.toUppercase at (REPL:1:1).
Could not find method toUppercase in class java.lang.String

This is fixed in alpha11

Clojure 1.12.0-alpha11
user=> (map String/.toUppercase ["foo" "bar"])
Syntax error (IllegalArgumentException) compiling fn* at (REPL:1:1).
Error - no matches found for instance method toUppercase in class java.lang.String

Oh, crikey. I mixed up the alphas. Thanks!

the basic idea behind this in deps.edn is that conveying arbitrary edn on the command line is fraught and given that we have a configuration file, you should be able to convey arbitrary data there by giving it a name (the alias). this can be used by tools, or even your app, in particular via the new clojure.java.basis api added in 1.12

alex.miller@alex ~ % clj -Sdeps '{:aliases {:foo ["hi" "there"]} :deps {org.clojure/clojure {:mvn/version "1.12.0-alpha11"}}}'
Clojure 1.12.0-alpha11
user=> (require '[clojure.java.basis :as basis])
nil
user=> (-> (basis/current-basis) :aliases :foo)
["hi" "there"]

oh cool, that makes sense.

tools that wish to leverage this as a config mechanism can do so, but are strongly encouraged to use namespaced alias keys

the more typical use of aliases as information for command execution by the CLI is a self-serving use of this - these are aliases conveying data for use by tools.deps in resolving the classpath

i see that a section has been added to the docs: https://clojure.org/reference/clojure_cli#_using_aliases_for_custom_purposes - that's helpful

once 1.12 releases, your example above would be a good demonstration

that said, I don't know how Sean is using them, so I'm sure he will add that context :)

Mornin' 🙂 My external test runner for Polylith lets you use aliases-as-data for passing JVM options to the subprocess: https://github.com/seancorfield/polylith-external-test-runner/?tab=readme-ov-file#passing-jvm-options and here's how we use that at work:

:poly-test-jvm-opts ["--enable-preview"
                       "-client"
                       "-Dclojure.core.async.go-checking=true"
                       "-Dclojure.spec.check-asserts=true"
                       "-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
                       "-Djdk.httpclient.allowRestrictedHeaders=host"
                       "-Dlog4j2.configurationFile=/var/www/worldsingles/development/resources/log4j2-silent.properties"
                       "-Dlog4j2.formatMsgNoLookups=true"
                       "-Dlogged-future=synchronous"
                       "-Duser.timezone=UTC"
                       "-Dws.pubsub.buffer-size=10000"
                       "-XX:-OmitStackTraceInFastThrow"
                       "-XX:+TieredCompilation"
                       "-XX:TieredStopAtLevel=1"]
  :poly {;...
                polylith/clj-poly {:mvn/version "0.2.19"
                                   :exclusions [org.slf4j/slf4j-nop]}
                io.github.seancorfield/polylith-external-test-runner
                {:git/tag "v0.4.0" :git/sha "eb954fe"
                 :deps/root "projects/runner"
                 :exclusions [org.slf4j/slf4j-nop]}
   :jvm-opts ["--enable-preview"
              ...
              ;; for the subprocess that poly test invokes:
              "-Dpoly.test.jvm.opts=:poly-test-jvm-opts"]

...and then in the runner itself:

java-opts      (or (System/getenv "POLY_TEST_JVM_OPTS")
                           (System/getProperty "poly.test.jvm.opts"))
        opt-key        (when (and java-opts (re-find #"^:[-a-zA-Z0-9]+$" java-opts))
                         (keyword (subs java-opts 1)))
        java-opts      (if opt-key
                         (into [] (remove nil?) (chase-opts-key (get-project-aliases) opt-key))
                         (when java-opts (str/split java-opts #" ")))

(defn- get-project-aliases []
  (let [edn-fn (juxt :root-edn :project-edn)]
    (-> (deps/find-edn-maps)
        (edn-fn)
        (deps/merge-edns)
        :aliases)))

(defn- chase-opts-key
  "Given an aliases set and a keyword k, return a flattened vector of
  options for that k, resolving recursively if needed, or nil."
  [aliases k]
  (let [opts-coll (get aliases k)]
    (when (seq opts-coll)
      (into [] (mapcat #(if (string? %) [%] (chase-opts-key aliases %))) opts-coll))))

wow, that's quite a bit of code. makes sense tho, you're juggling a lot of stuff.

sean@sean-win11-desk:~/workspace/wsmain$ find . -name deps.edn|wc
    210     210    7226

210 deps.edns? holy shit

Our top-level deps.edn is > 600 lines and our build.clj is > 400 lines. Polylith has a deps.edn file per "brick"" (`bases` for applications, components for reusable pieces) plus one per "project" (the actual thing you build from those bricks). 145K lines of code total.

that's very cool

A thing to consider: https://stackoverflow.com/questions/15756075/is-it-true-that-having-lots-of-small-methods-helps-the-jit-compiler-optimize

the limits are really stack memory, not a fixed limit. it's pretty unusual to hit a stackoverflow unless you're in some recursive process

in general, I wouldn't make refactoring decisions based on stack size

note that assoc-in and other core functions are coded in a way that they could stack overflow.

setting=> (some? (assoc-in {} (repeat 50000 :a) :hi))
Execution error (StackOverflowError) at (REPL:1).
null

Depends. It's silly in small cases, for largely recursive algorithms that you don't actually need a stack for, it wouldn't be silly. I would say though what's not silly is designing the code so it's not as deep. It's easier to move things horizontally then up/down. It's also easier to follow a series of steps, than lots of inner branching.

To answer your question more directly. You can set the stack memory as a JVM arg when you launch the app. I think default is 1mb, but it depends on JVM, their version, what operating system it runs on, etc. they might all have different default.

I think the option is -Xss for stack size

Each thread gets a stack. So it's per-thread. And if you max it out, you get StackOverflow

Misc well-known Clojure libraries tend to need a higher stack (`-Xss` as mentioned above) to properly work Some people have a hard time accepting that. Personally I have it clear - Java algos and Clojure algos are vastly different, the JVM / its defaults were made primarly with Java in mind > I feel a slight discomfort because I don't like growing my call stack. Considering even the simplest function in Clojure has a quite high stack depth already, it barely makes a difference What I mean is, if I hit Ctrl+\ in my repl I can easily see many layers like:

at clojure.core.protocols$fn__8244.invokeStatic(protocols.clj:136)
	at clojure.core.protocols$fn__8244.invoke(protocols.clj:124)
	at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
	at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)

I've never heard of a Clojure library needing a different -Xss

clojure.algo.monads, Meander (iirc)

news to me

Ya same, never had to change the Xss config on any prod system or personal stuff either. I've never used those libs though, but, it sounds like they have unbounded recursion that consumes the stack? If so, no Xss setting will work, you just have to rewrite the implementation. If they just have a need for higher Xss but it doesn't grow with input size, that's weird. I can't imagine what they're doing that's so deep

Like, to stack overflow you'd need to go beyond 1000 depth, that's quite large.

Thanks all for the interesting replies

To @U47G49KHQ: it is important to understand that by default, without https://clojure.org/reference/compilation#directlinking, Clojure functions cannot be inlined because of Var indirection. So, function calls in Clojure are not as "free" as they are in Java.

Second point: A well-written Clojure program usually has a much cleaner call stack than a framework-driven Java program. Case in point:

Generally when people release their app though, they will enable direct linking, and others I feel.

Is direct linking a common option set during AOT? I’m not sure we’ve ever done it and I don’t hear about it much.

Generally when people release their app though, they will enable direct linking, and others I feel.To be honest, I really doubt that. It is a great feature, but it is still not default, obscure, and comes with its own set of tradeoffs.

Oh well, might just be me then. I always AOT, elide-meta and direct-link for release. Not for libraries, but for apps/services yes.

Only downside as far as I know is you can't redefs which prevents repl usage.

Well, it doesn't prevent it. Just limits your ability to redef and have what was already calling that function be updated to call the new function

have you checked how much performance you get from direct linking? I wonder if the benefits outweigh the loss of repl functionality

No I haven't. Just kind of did it cause free boost. We never redef in production by policy. We even try to rarely REPL-in, but when we do we use it to debug. The risk of breaking the node more than it is in prod is too high with redef, you can fat finger easily and all.

yeah. i would never open a repl against a real running instance, but i like being able to debug the actual production jar

java "$(socket-repl 6006)" -jar real-jar.jar is lovely

It's a good idea to do all REPLing in a sidecar process. It shouldn't be too hard to create two artifacts and two processes (one direct-linked, the other not)

> Only downside as far as I know is you can't redefs which prevents repl usage. Yes, it is pretty much the only downside, but still massive for me. I asked before for the ability to control what's direct linked: https://ask.clojure.org/index.php/12488/finer-grained-control-over-direct-linking

You do have the ability to control it. You can mark it with ^:redef to avoid direct linking.

Or you meant like, opt-in instead of opt-out ?

I mentioned that in the question.

Ah ok, ya I just read it

In my backed-only-by-intuition assumption, enabling direct linking just for clojure.core functions and nothing else would give 80-90% of the performance benefits.

> However, if the developer has DL disabled, then the calls to clojure.core functions would go through Var resolution > Is this true? I thought it would use the precompiled classes first, and those are direct linked.

Honestly, this can be a lib. Pretty sure you can walk your code and add the annotations to all vars as a build step. Then you could specify namespaces to include/exclude.

> Is this true?

user=> (decompile (not (empty? [])))

...
    public static void load() {
        ((IFn)cjd__init.__not.getRawRoot()).invoke(((IFn)cjd__init.__empty_QMARK_.getRawRoot()).invoke(PersistentVector.EMPTY));
    }

    public static void __init0() {
        __not = RT.var("clojure.core", "not");
        __empty_QMARK_ = RT.var("clojure.core", "empty?");
    }
...

> Honestly, this can be a lib. Pretty sure you can walk your code and add the annotations to all vars as a build step. Then you could specify namespaces to include/exclude. I'd rather patch the compiler if I wanted to go that route. Would be more robust.

Ya, but that type of PR will just linger forever lol. I'm saying it more from a practical standpoint, not that I think it's better as a lib 😛

I meant as in "patch the compiler locally and use it instead of mainline".

Oh I see. Well, sure. But then you can't offer it to the world and you need to keep rebasing

I feel using rewrite-clj it should be not too hard to do

It won't work for libraries then, or you have to be content with not being able to redefine functions in libraries. Which may be fine, but then, perhaps the blanket :direct-linking true will be fine.

Ya, if the lib is direc-linked, like Clojure it will. You could also, more annoyingly, do it for libs. You'd need to unjar and rejar though.

I see, don't even have to rejar, just unpack onto the filesystem and compile from there.

Why would you want libs not direct-linked though?

I can't remember the last time I redefed an inner lib function that my code doesn't call

You have to debug weird stuff sometimes:). It doesn't happen often, but when it does, you sure as hell want that ability. But again, really depends on what you are doing. In a really serious type of application where data corruption may be critical, I'd stay away from redef for sure

Ah, I think I'm wrong about the Clojure core lib being AOTed

That's why direct-linking depends on if you have it on or off

But then, what does it mean for Clojure lib to be direct-linked?

It means that functions in clojure.core invoke direct-linked variants of other functions in clojure.core

> As of Clojure 1.8, the Clojure core library itself is compiled with direct linking. This is from: https://clojure.org/reference/compilation#directlinking

If I set direct-link to true when I compile, won't everything, including all my dependencies, be direct-linked? So like, it's not like Clojure has to explicitly do anything for it to be direct-linked if I set direct-link to true no?

Yes. But if you do not enabled direct-linking (which is the default state), clojure.core functions calling other clojure.core functions will still be direct linked.

Here's clojure/core$frequencies.class (from the Clojure jar) decompiled:

public static java.lang.Object invokeStatic(java.lang.Object coll) {
        java.lang.Object object = coll;
        coll = null;
        return core.persistent_BANG_.invokeStatic((java.lang.Object)core.reduce.invokeStatic((java.lang.Object)new fn__8614(), (java.lang.Object)core.transient.invokeStatic((java.lang.Object)clojure.lang.PersistentArrayMap.EMPTY), (java.lang.Object)object));
    }

So Alex's answer to your question is outdated? He said that fastmath for example wants to redef core functions no? So does that mean you can no longer do that?

No, he's correct. You can redefine clojure.core functions for your code (or any code that doesn't use direct linking)

Ok, but it won't redefine the function that clojure.core functions call on each other

You can redefine clojure.core/persistent! so that your code uses the new version, but clojure.core/frequencies will still use the old version of persistent!

Interesting limitation. So you could not have fastmath detect boxed calls a clojure function makes by calling other math operations internally

Unless you deleted the class files from the clojure jar

Theoretically, you can put a sources-only clojure.jar onto the claspath, with file change date that is later that in the real clojure.jar. Not really robust, but that will force a clojure.core recompile.

I actually feel like, it would be nice to lean into direct-linking even more. I would wrap more things in a Var indirection for better REPL reloads. And then rely more on direct-linking to elide. And then it could also make sense to have better support for granular config of what to direct-link or not. Like, hum, now I forgot haha. What are the things you can't redef. Protocols?

Found a comment

;; Decoding remains the same, you are responsible for doing custom decoding.

Looking at what I did in a similar situation here is only add-encoder, once you encode something you lose type information, so it only becomes a String/Number/JSObject so you would lose the ability to know what to decode it to.

I got my answer!

for Java 8 time classes I think you can just do something like this for encoding:

;;; add simple string json encoders for Java 8 classes
(json/add-encoder Duration json/encode-str)
(json/add-encoder Instant json/encode-str)
(json/add-encoder LocalDate json/encode-str)
(json/add-encoder LocalDateTime json/encode-str)
(json/add-encoder LocalTime json/encode-str)
(json/add-encoder MonthDay json/encode-str)
(json/add-encoder ZoneOffset json/encode-str)
(json/add-encoder ZoneId json/encode-str)
(json/add-encoder OffsetDateTime json/encode-str)
(json/add-encoder OffsetTime json/encode-str)
(json/add-encoder Period json/encode-str)
(json/add-encoder Year json/encode-str)
(json/add-encoder YearMonth json/encode-str)
(json/add-encoder ZonedDateTime json/encode-str)

I was able to do something a lot simpler with clojure.data.json

(defn encoder-fn
  "Custom encoder on java.time.Duration types"
  [key val]
  (cond
    (= (type val) Duration) (str ":java.time.Duration" "[" val "]")
    :else val))

(json/write-str {:a 1 :b (java.time.Duration/ofSeconds 1)} :value-fn encoder-fn)

(defn decoder-fn
  "Custom decoder on java.time.Duration types"
  [key val]
  (cond
    (str/starts-with? val ":java.time.Duration")
    (Duration/parse (second (str/split val #"\[|\]")))
    :else val))

(json/read-str "{\"a\":1,\"b\":\":java.time.Duration[PT1S]\"}" :key-fn keyword :value-fn decoder-fn)

Fwiw if you’re dealing with structures of a known shape, malli is great for this sort of thing. Adding these sorts of tags isn’t all that different from what transit does too

Oh good to know! I can look into that too