Fork me on GitHub
#tools-build
<
2022-05-04
>
imre17:05:46

I'm experimenting with b/compile-clj to a) reduce repl/script startup time b) compile stuff before it goes into an uberjar (so the app starts up quicker when ran from the uberjar) My root assumption (and please correct me if this is wrong) is that in order to make startup/warmup as quick as possible I should try and compile as many of the namespaces that need to be loaded until the app starts up and reaches a warm state so all of that loading happens quickly. The part of the interface of b/compile-clj that deals with including namespaces in the compile.clj script is either a fix collection of namespaces or a collection of paths (provided by the caller or extracted from the basis), which is then resolved through tools.namespace to a collection of namespaces. As I'm looking to maximize the number of namespaces to compile, I'm planning on using this latter option. However, a lot of loading in my app(s) happen dynamically, using functions like (require) and (requiring-resolve), which means that if I just provide my app's own source paths, tools.namespace won't find all the namespaces my app depends on at runtime (as it only parses ns declarations). What I tried therefore is using all the paths from (:classpath-roots basis) but this resulted in compile.clj also including namespaces I do not want to compile as I know they are not needed for my app (some of them even result in compilation failures, like clj-kondo hooks distributed with some libs). I'm thinking a solution would be to instruct b/compile-clj to discover namespaces from the entirety of classpath-roots, then make it filter the resulting namespace collection (remove the ones I don't need) before those get into compile.clj. Unfortunately I don't think this is currently an option. Has anyone run into something like this before? I'm now thinking of using a modified version of compile-clj but I'd be interested to know whether others have other suggestions. Or perhaps something telling my why I shouldn't be doing this at all. Thank you.

seancorfield17:05:55

I don't have an answer but I will point out that plain require is not thread-safe if you use it heavily at runtime, but requiring-resolve is thread-safe.

seancorfield17:05:44

There's an undocumented (and probably private) function called serialized-require in clojure.core that was introduced at the same time that is the thread-safe version of require.

seancorfield17:05:08

(we use that at work to load namespaces dynamically in one of our apps)

imre17:05:49

Thank you, I think it's mostly requiring-resolve, but worth having a look over.

seancorfield17:05:39

It used to bite us occasionally before we updated to Clojure 1.10 (and switched to serialized-require).

imre15:05:58

In case someone also bumps into this and stumbles across this thread, there's now a related clj-kondo issue: https://github.com/clj-kondo/clj-kondo/issues/1685