Fork me on GitHub
#tools-deps
<
2022-10-27
>
Noah Bogart15:10:49

The uberjar output by b/uber is much bigger than the one created by leiningen. is there any way to determine the source of the difference? (If this is something to ask in #C0AB48493 I can move the question over as well.)

seancorfield16:10:18

Was your uberjar AOT'd with Leiningen? Perhaps that's the difference? Either way, I'll be interested to hear what comparing the contents of the two JARs reveals...

Noah Bogart16:10:52

it was, we have :aot :all in the :uberjar profile.

borkdude15:10:13

you can unzip it and see what's in there

borkdude15:10:26

zipinfo ~/.m2/repository/io/github/borkdude/SnakeYAML/1.33/SnakeYAML-1.33.jar

borkdude15:10:37

(from my bash history)

Noah Bogart16:10:29

Thanks for the suggestion, I'm looking now with zipinfo and seeing that 1) all of the dependency source files are included (.clj, .java) and 2) certain dependencies are included as well. makes me wonder how leiningen did it. For example, we use fundingcircle/jackdaw, which has a dependency on aleph. the tools uberjar has aleph/flow.clj etc and both aleph/utils/RequestTimeoutException.java and compiled aleph/utils/RequestTimeoutException.class files, whereas the leiningen doesn't have any files related to aleph in it.

seancorfield16:10:06

How does jackdaw declare the dependency on aleph? I wonder if it assumes a Maven scope that tools.build/t.d.a don't understand?

seancorfield16:10:56

Hmm, nope, it's a regular dependency... I would expect Leiningen to just add all that in...

seancorfield16:10:01

It makes no sense for it to be excluded from an uberjar if it is a dependency. An uberjar needs everything so it can be run.

seancorfield16:10:38

Perhaps there's something in your project.clj that is telling Leiningen to exclude some of that stuff from the uberjar for some reason?

Alex Miller (Clojure team)16:10:21

and it is possible to do the same by creating a modified basis that excludes a dep in tools.build if you want that. not sure why dependency sources would be included unless you're doing something to pull the source classifier jar

dpsutton17:10:02

leiningen would traditionally make two jars. are you sure you are comparing the uberjar from lein to the uberjar from tools.build?

dpsutton17:10:27

I made a new lein project, added aleph to it and ran lein uberjar

❯ ls -l target/uberjar
total 18032
drwxr-xr-x  4 dan  wheel      128 Oct 27 12:09 classes
-rw-r--r--  1 dan  wheel  8501590 Oct 27 12:09 hi-0.1.0-SNAPSHOT-standalone.jar
-rw-r--r--  1 dan  wheel    13802 Oct 27 12:09 hi-0.1.0-SNAPSHOT.jar
drwxr-xr-x  3 dan  wheel       96 Oct 27 12:09 stale
note there are two jars of vastly different sizes

Noah Bogart17:10:48

Aha, I was looking at the wrong one. Thanks for the help, everyone

👍 1
seancorfield17:10:07

Now I'm curious how much difference in size there is (or not) between the actual uberjar from Leiningen (not the library JAR!) and the uberjar from tools.build??

Noah Bogart17:10:56

Using zipcmp:

$ zipcmp -i muncher-0.1.0-SNAPSHOT-standalone.jar muncher-standalone.jar 
--- muncher-0.1.0-SNAPSHOT-standalone.jar
+++ muncher-standalone.jar
...
-       1450 faa8953c buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012$fn__63013$fn__63014.class
-       2704 dda4f5d4 buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012$fn__63013.class
-        841 485e6fe9 buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012.class
-       1882 6785282a buddy/core/bytes$concat$iter__63004__63010$fn__63011.class
-        783 ac02f762 buddy/core/bytes$concat$iter__63004__63010.class
+       1450 3ed59749 buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079$fn__63080$fn__63081.class
+       2704 00c515eb buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079$fn__63080.class
+        841 e78f36c7 buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079.class
+       1882 fd25a2e6 buddy/core/bytes$concat$iter__63071__63077$fn__63078.class
+        783 97cf0ea9 buddy/core/bytes$concat$iter__63071__63077.class
+        984 954a77bf buddy/core/bytes$concat.class
-        984 a46bbf08 buddy/core/bytes$concat.class
-       2067 2450f6ba buddy/core/bytes$equals_QMARK_.class
+       2067 621de668 buddy/core/bytes$equals_QMARK_.class
-       3564 6024f654 buddy/core/bytes$fill_BANG_.class
+       3564 d744d4cf buddy/core/bytes$fill_BANG_.class
-       1758 dc20a963 buddy/core/bytes$fn__62989.class
+       1758 00eb8eeb buddy/core/bytes$fn__63056.class
-       2292 98b7ccfc buddy/core/bytes$loading__6789__auto____62987.class
+       2292 ecfbc801 buddy/core/bytes$loading__6789__auto____63054.class
+       6129 7c316f0d buddy/core/bytes__init.class
-       6129 ec4dd1d6 buddy/core/bytes__init.class

dpsutton17:10:13

what about the size difference? that’s comparing things you wouldn’t expect to be identical. But if that’s the diff (and its just counters in the compiler) you are golden

seancorfield17:10:51

Looks like the sizes are all identical there.

Noah Bogart17:10:16

the output .class files look to be the (roughly) same, maybe a byte of difference in the occasional file

seancorfield17:10:30

(Clojure compilation is not a reproducible process, i.e., you don't get identical byte code each time you compile the same code but you get equivalent byte code -- those differences in generated classes are just that)

dpsutton17:10:56

i’m interested in the size of the two jars. if they are close then no problemo

Noah Bogart17:10:16

$ exa -la
.rw-rw-r-- 160M noah 27 Oct 11:24 muncher-0.1.0-SNAPSHOT-standalone.jar
.rw-rw-r--  31M noah 27 Oct 11:23 muncher-0.1.0-SNAPSHOT.jar
.rw-rw-r-- 159M noah 27 Oct 11:21 muncher-standalone.jar

seancorfield17:10:35

And I thought some of our uberjar files were big! 🙂

Noah Bogart17:10:22

We don't do any splitting at all, so this is our entire 63k lines of clojure backend lol

seancorfield17:10:05

(our largest one is about 79M at this point -- but we've been working hard to keep them small... most of ours are around 35M... about 18 uber JARs from 130K lines of Clojure)

clapping 1