Fork me on GitHub
#leiningen
<
2022-12-20
>
jumar13:12:31

I'm trying to build an uberjar without sources. The :omit-source isn't enough, as I learned here: https://github.com/technomancy/leiningen/issues/1357 However, the https://github.com/technomancy/leiningen/issues/1357#issuecomment-27005055 with :uberjar-exclusions doesn't seem to work for me. This is my uberjar profile

:uberjar {:aot :all
          :omit-source true
          ;; :omit-source isn't enough;
          ;; make sure to omit sources from dependencies like analysis lib: 
          :uberjar-exclusions [#"\.(clj|java)"]}
The trouble is that the JAR fails to start because of java.io.FileNotFoundException: Could not locate clojure/core/specs/alpha__init.class, clojure/core/specs/alpha.clj or clojure/core/specs/alpha.cljc on classpath
at clojure.lang.Util.loadWithClass(Util.java:251)
        at my.main.<clinit>(Unknown Source)
Caused by: Syntax error macroexpanding clojure.core/extend-protocol at (0:0).
        at clojure.lang.Compiler.checkSpecs(Compiler.java:6989)
        at clojure.lang.Compiler.macroexpand1(Compiler.java:7005)
        at clojure.lang.Compiler.macroexpand(Compiler.java:7092)
        at clojure.lang.Compiler.eval(Compiler.java:7178)
        at clojure.lang.Compiler.eval(Compiler.java:7149)
        at clojure.core$eval.invokeStatic(core.clj:3215)
        at clojure.core$eval.invoke(core.clj:3211)
        at clojure.core.matrix.impl.persistent_vector__init.load(Unknown Source)
        at clojure.core.matrix.impl.persistent_vector__init.<clinit>(Unknown Source)
        ... 574 more
Caused by: java.io.FileNotFoundException: Could not locate clojure/core/specs/alpha__init.class, clojure/core/specs/alpha.clj or clojure/core/specs/alpha.cljc on classpath.
        at clojure.lang.RT.load(RT.java:462)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.lang.Compiler.ensureMacroCheck(Compiler.java:6975)
        at clojure.lang.Compiler.checkSpecs(Compiler.java:6987)
        ... 582 more

Rupert (All Street)14:12:17

Please try unzipping the jar file and looking inside of it. You will have to do this anyway to ensure that you have successfullly removed the source code. • Without :omit-source and without :uberjar-exclusionsWith :omit-source and without :uberjar-exclusionsWith :omit-source and with :uberjar-exclusions

Rupert (All Street)14:12:54

Once unzipped you can use tree command to list files and maybe compare the output using diff --color

jumar14:12:38

I did that already multiple times. This is how I found plain :omit-source doesn't work (it contains both class files and source code) There are so many clj files but I guess I need to have a look at clojure/core/specs and see what's in the sources that doesn't get compiled..

Rupert (All Street)14:12:18

Yes - focus on clojure.core to start off.

Rupert (All Street)14:12:32

Possible approach: • If you have internal libraries then you can run aot + omit source on them first when you compile/install them. • Then build your application uberjar with aot + omit-source to generate the final uberjar then none of your own source code will be in the uberjar but there will be code for open source libraries that may not matter to you.

jumar14:12:59

I considered that option but didn't want to complicate the build. Then I didn't know exactly how to do it O:-).

Rupert (All Street)14:12:16

> This is how I found plain :omit-source doesn't work (it contains both class files and source code) For code in your application project or libraries?

jumar14:12:26

Comparing two versions of the jar (omit-source without exclusions VS omit-source with exclusions) core specs seems to be missing

# this is in the uberjar without exclusions
ll clojure/core/specs/alpha.clj
-rw-r--r--  1 jumar  staff   7.7K Sep 14  2021 clojure/core/specs/alpha.clj

# this directory in the built uberjar is empty
ls clojure/core/specs/

Rupert (All Street)14:12:12

Yeah - so it looks like it's uberjar + aot is not going to aot compile all of your dependencies. Then your exclusion is removing the required clj files.

Rupert (All Street)14:12:04

I think you can just add :aot true + :omit-source true to all of your internal libraries + the final application. That way the final uberjar will have none of your code but will have open source code. Would that be ok for your usecase?

jumar14:12:03

Yes, I need to try that. But as a special profile or something because I don't want to do that during normal (dev) workflow. Do you know what's the easiest way to do it?

Rupert (All Street)14:12:56

I have a feeling that approach will work although haven't done it in a while.

jumar14:12:58

I mean what we do right now is simply

cd A && lein install
cd ../B && lein install
cd ../C && lein uberjar

Rupert (All Street)14:12:37

Yup - so process is the same - just adding :aot :all+ :omit-source true to each project.clj file.

jumar14:12:50

Good, I'm gonna try it. Thanks for your help!

mikerod22:12:06

@U06BE1L6T when you do :aot you should automatically AOT all of your dependencies and transitive ddeps - this would include clojure.spec stuff. You should then be able to just do the :uberjar-exclusions you did before. I don’t see why that wouldn’t work. I don’t see why you’d need a setup any harder than putting these extra options in a profile you use only for uberjar’ing

:uberjar {:omit-source true :aot :all :uberjar-exclusions [#"\.(clj|java)"]}

mikerod22:12:42

Clojure AOT wil aggressively compile all clj files that are encountered on any use/`require`/`load` call. If you are using a lein version newer than 1.4.1 then the option :clean-non-project-classes should be false/off by default - meaning it all AOT class files will remain in your compile path (and get into uberjars). https://github.com/technomancy/leiningen/blob/2.10.0/NEWS.md#141--2010-12-16

jumar05:12:40

@U0LK1552A Yes, that's what I hoped for. However, it breaks on core specs for some reason. I'm really using the uberjar config that I pasted above - the same one as you stated here:

:uberjar {:omit-source true :aot :all :uberjar-exclusions [#"\.(clj|java)"]}
But it just doesn't work 😞 If it matters, I'm using this version:
Leiningen 2.9.10 on Java 1.8.0_302 OpenJDK 64-Bit Server VM

mikerod05:12:28

I’m guessing you are definitely loading the missing spec ns at some point?

mikerod05:12:04

If not you’d have to add it to :aot vector

jumar05:12:57

I'm not sure I understand. What do you mean by loading the missing spec ns? We use spec in many places and require it as needed. However, I don't think we ever reference directly clojure.core.specs.alpha

mikerod06:12:29

As long as it required transitively somehow. Otherwise you wouldn’t get an AOT classfiles produced

jumar06:12:54

I tracked it down to the use of core.matrix library: https://github.com/mikera/core.matrix I created a minimal reproducer here: https://github.com/jumarko/uberjar-sample/blob/master/app/README.md I don't understand in detail how clojure.core.specs.alpha work but I noticed it's loaded automatically in Compile.java https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/jvm/clojure/lang/Compile.java#L58

jumar06:12:31

I even tried to add an explicit require for clojure.core.specs.alpha but it didn't help: https://github.com/jumarko/uberjar-sample/commit/3e3e63db7ab48f49130d8a386d8f733e7c3b65a6

jumar07:12:29

I had to whitelist clojure/core/specs/alpha.clj to make this work: https://github.com/jumarko/uberjar-sample/commit/6d26f3cceb48b2ab2acc0b6553cc1db8eb49b580

:uberjar-exclusions [#"(?<!clojure/core/specs/alpha)\.(clj|java)"]

Rupert (All Street)08:12:16

That looks like a good solution.

jumar10:12:38

It's been unfortunately much more complicated in the real project, due to the usage of duct/integrant, ring params middleware and more. This stuff loads namespaces dynamically and simply breaks if sources are ommited. I'm in the process of excluding only specific set of namespaces instead of trying to exclude everything...

Rupert (All Street)12:12:53

Maybe try adding a big (:require [.....])statement to your main namespace listing all of the namespaces that you are likely to end up loading dynamically. Obviously this may not be ideal for some situations.