Clojurians Log v2

@orestis have you looked at pack.alpha?

> What I’m after is essentially creating a docker layer with all my (compiled) dependencies, that should, in principle, not changing very often, and another docker layer with all my (compiled) source. Aim is to deploy to production and get fast startup time (hence the compiled requirement). The split of dependencies & application source is exactly what pack takes care of. It doesn’t get involved with compilation though. Splitting the AOT compiled stuff is really difficult as discussed above.

I’m not sure there’s a term of art for that except for (javaee) container-specific deploys (where the container server provides the deps, and WAR files are the application). I think typically when you start separating, you just make a lib directory of dependency jars instead of dependency + application uberjars, because it’s much simpler and safer.

there’s also jar-of-jars, but that’s just another uberjar-like technique to get one self-contained jar

What I'm after is essentially creating a docker layer with all my (compiled) dependencies, that should, in principle, not changing very often, and another docker layer with all my (compiled) source. Aim is to deploy to production and get fast startup time (hence the compiled requirement).

if these dependencies you want to be slower-moving are clojure-aot compiled, I think there may be some trickiness there

clojure compilation is transitive to all required code, there is no stable clojure ABI, and I’ve heard can be hard to predict what the output class filenames can be

IOW if this is clojure code, I don’t think you can easily and safely separate the “my aot-ed code” from the “this lib’s aot-ed” code the same way you can with java lib class files

but you could separate by clojure source or java classfile depencency jars vs application clojure source jars

but if you clojure-aot, you probably shouldn’t split up the result of the compile into multiple jars or allow the same clojure source to have aot code on the classpath.

I thought clojure AOT means just generating a .class file per .clj file?

there’s not a straight 1-1 correspondence between class files and clj files

I would push all the built dependencies to a repository, as part of the CI workflow of each project. Then in the project that uses the dependencies the docker image can include those dependencies. The cache would be used unless a newer version of that dependency is pushed to the repository Something like Artefactory or AWS artefact repository

hm, 1-1 between class file and namespace?

no

try it 🙂

this is why clojure libs don’t distribute jars with class files

https://clojure.org/reference/compilation

Ah, I was thrown by the usual cannot find foo__init.class for the foo namespace.

I still think that AOT compiling Clojure dependencies today and using them with AOT compiled Clojure source tomorrow should work, as long as we're talking about the same Clojure/JVM version.

It may work most of the time, but really you’re not meant to combine compilation output from different processes running different classpaths

or, you’re on your own, you may encounter subtle problems

it’s not impossible, but I wouldn’t trust myself to do it right

Here’s a discussion where Colin Fleming (who made Cursive) asks about using bits of aot-compiled stuff together (for incremental compilation in his case.) The thread touches on some of the challenges in more detail. https://clojurians.slack.com/archives/C06E3HYPR/p1651097379296839

This is the key:

> clojure basically behaves like some kind of whole program optimizing compiler, despite not doing any of things those do > [5:19 PM] so like taking apart the output of an optimizing compiler and linking it with something compiled separately is tricky, so is doing that with aot compiled clojure

Well I can certainly compile everything together and split the output files. If Docker hashes them and the output is the same, problem solved

compilation output is not deterministic

probably never going to hash the same

You can put the contents of .m2 after a clojure -P into your slower-moving layer and probably get most or all of the savings from not having to download dependencies separately, but that doesn’t include the trickier AOT part

or slightly fancier, a tools-deps program that copies all basis jars into a lib directory

then put the lib in a layer

I’m already doing the Clojure -P and Docker is smart to cache that automatically. Definitely helps to not download the universe. My quest for smaller layers is insignificant compared to that.

AOT compilation is a basic requirement for us, for faster startup times, so if we can't split AOT-compiled classes that's a no go.