Clojurians Log v2
tools-build
A small point on the docs and expectations for tools.build / uber.
Yesterday I was helping a colleague get started with tools.build; and it’s not 100% clear that resources you want to put in the jar from your project come via :class-dir only.
i.e. logically there are two potential sources for this information: :class-dir and the :basis. We both understood this, but it wasn’t immediately obvious which was used where.
It seems after a little bit of digging that the basis is only used by uber for discovering what libs to put into the jar, and that you essentially need to copy additional resources into :class-dir yourself.
After discovering this there is a subtle hint that this is the case in the doc string:
> Create uberjar file containing contents of deps in basis and class-dir.
However I wonder if it might be worthy of a more explicit mention, given that we both missed this subtlety either in the guide, doc string or somewhere else.
Sure, which part needs elaboration? basis for deps OR put stuff in class-dir. or both?
updated at https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-uber - let me know if that covers it
@alexmiller: Firstly sorry for the delayed reply…
For me I knew that the basis would tell uber what libs to package; I didn’t know uber only used it for that.
I think your expanded doc string is fantastic, and for me totally nails it! I’ll run it by my colleague and see what she thinks too.
🙇
Her comments: > Nice, that’s much more helpful! > How incredibly efficient
(and now a big discussion about how great the clojure community is)
happy ubering :)
😁
another question; any idea what makes uber so slow? It seems slower than it should be. I know it does a lot of extracting and repackaging, and I guess it took a long time under leiningen too; but it seems surprising how slow it is.
Hmmm… I wonder if it’s because you’re using the old java.util.zip stuff; rather than the newer java.nio.FileSystems API.
I’ve written some code with the nio FileSystems API that copies large and small files in and out zips relatively recently and anecdotally it’s pretty quick
I don’t have any concrete numbers; but I could probably get some
hmm I guess JarInputStream and JarOutputStream aren’t available under nio
Sean, you are correct. The issue lies mostly with my corporate laptop antiviral scanner. The tools.build/uber slowdown is magnified due to the higher IO. Without the scanning active, my times are more comparable:
-depstar: 12s
-tools.build: 50s
When running it from a personal laptop, the times are 6s and 13s, respectively. So this appears to be local issue as a result of high disk IO.
@sashton Thanks for following up with your analysis. Yeah, that makes much more sense and those times are closer to what I would expect with a decent HD and no system interference. Anti-virus software generally seems to do more harm than good 🙂
@rickmoynihan have you tested it in isolation, excluding startup time and compile-clj?
It was at a REPL in the build env
right
calling just uber
(b/uber {}) ?
well we had some project aliases too in the basis, and a :class-dir etc
IIRC the final jar was about 150mb — I don’t have it too hand to double check the details now though
quite big!
yeah I guess the dependencies/libs extracted are pretty big
depstar used to work like tools.build (copy everything to a temp folder, and then JAR it all up) and in 2.0 I switched to copying directly into a FileSystem JAR file and it was a massive speedup, like more than double. But uber unpacks every dependency down to the file system, and then reads all of that plus the source and classes etc, back from the file system as it builds the JAR, so it's doing a lot more work.
It's a much simpler approach, and has the benefit of leaving the working directory intact afterward if you need to debug things, but it is certainly a slow approach.
Hey @seancorfield. I’m reading up on this old thread as I’m also observing slow uberjar times using clojure.tools.build.api. I compared uberjar-ing the same project with both tools.build and depstar and I see a significant speed difference:
clojure.tools.build.api/uber:
$ time clj -T:build uber
real 5m14.577shf.depstar.uberjar/build-jar:
$ time clj -T:build uber
real 0m30.601sdepstar in favor of tools.build. Did you see a similar drop in build times when you switched off of depstar?Yes, tools.build is going to be slower because you copy everything into a target folder and then build the uberjar from that.
I'm surprised the difference is 5 minutes compared to 30 seconds -- so I expect you are not comparing apples to apples there.
I would expect depstar to "only" be perhaps twice as fast as tools.build if you are doing identical operations since the only thing it does differently is not copying files to an intermediate folder.
We saw maybe 10-30 seconds extra time using tools.build over depstar I think (our JARs are 25-70MB).
I notice that when adding :paths ["."] to the :build alias, clj -A:build gives you a REPL where you can require 'build and do stuff. Why doesn't it work without it? What's the logic that makes clojure -T:build see build.clj ? -T implicitly adds . to the classpath?
Correct, I'm not happy with where this ended up wrt the repl so prob something more will be done there eventually
Imo it's a small price to pay to add :paths ["."] - it also makes it more explicit what's happening
We use clj -M:build -i build.clj -r as a way to get a REPL with the build.clj file loaded so we can run stuff interactively. Otherwise, my "main" development REPL is started with
SOCKET_REPL_PORT=5000 clojure -J-Dlog4j2.configurationFile=log4j2-sean.properties -M:rebel:portal:everything:dev:test:runner:build:dev/replbuild.clj but does not add . to :paths and then I can open build.clj in my editor and issue the "load file" editor command (so it doesn't need to be on the path, it just needs to get loaded into the REPL).That's also a way to do it