Fork me on GitHub
#tools-deps
<
2020-10-05
>
zetafish17:10:59

Hi all, I try to create an uberjar with depstar using the -X method. I.e. i updated my deps.edn to have an alias specifying a exec-fn and exec-arg. When running on my laptop clojure -X:uberjar a jar is generated just fine. However when I try to create the jar from a docker container it fails with

Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
-X:uberjar (No such file or directory)
The docker images I tried:
clojure:openjdk-11-tools-deps-1.10.1.469
clojure:openjdk-11-tools-deps-1.10.1.561

kirill.salykin18:10:59

it has interesting feature skinny jar

zetafish19:10:15

I want a self contained jar that I can run with java -jar app.jar so skinny does not fit the bill.

alexmiller17:10:48

you don't have the newest clj tools in your docker

alexmiller17:10:06

you need clojure 1.10.1.697

zetafish19:10:00

Thank you Alex. I presume the 697 image will become available on docker hub soon. In the meantime I built the image locally and the -X:uberjar works fine then.

alexmiller20:10:35

I don't have anything to do with the docker stuff, so no control over that for me

seancorfield17:10:39

@zetafish I'm planning an overhaul of my project's readme files this week to make that clearer (that some of the usage examples require 1.10.1.697). I also need to update all of the clj-new project stuff to generate projects that can work both ways.

dominicm18:10:07

Is clojure.libfile still present in 697?

seancorfield18:10:18

I don't see it in the system properties when running code via 1.10.1.697 @dominicm -- there's clojure.basis now, which is the path to the combined EDN file.

seancorfield18:10:42

I see it was still present in 590:

> /usr/local/Cellar/clojure\@1.10.1.590/1.10.1.590/bin/clojure
Clojure 1.10.1
user=> (System/getProperty "clojure.libfile")
"/Users/sean/.clojure/.cpcache/2731801087.libs"
user=> 

dominicm18:10:57

Ah, that could explain the bug I'm seeing. I thought it was being retained for backwards compatibility.

seancorfield18:10:27

@dominicm Lots of breakage between the previous stable and the current stable 🙂 but if you change (System/getProperty "clojure.libfile") to (some-> (System/getProperty "clojure.basis") (str/replace #"\.basis$" ".libs")) you can still get at the old file (although, as I recall, the basis file now contains the libs list as an element?)

seancorfield19:10:44

Yeah, (some-> (System/getProperty "clojure.basis") (slurp) (edn/read-string) :libs) should be the same data.

alexmiller19:10:28

yeah, I removed it in 697. we've never publicly doc'ed it and I was not aware of anyone using it (other than add-libs which won't work with latest anyways) so I didn't think there was any reason to retain it

dominicm19:10:52

I was using it to find myself on the classpath and remove my paths

alexmiller20:10:51

I'd say separating the tool from the basis is the better long term path for that

alexmiller20:10:11

we have an ongoing conversation about some details of that right now

dominicm20:10:00

If I could reliably identify my lib in the classpath, then that'd be fine 🙂. Happy to switch over to the newnew system when that comes out too.

dominicm20:10:15

A github search indicates I'm probably the only user anyway

dominicm20:10:30

We also use it in our internal tool based on the lib, but I can update that the same way

seancorfield21:10:48

The problem with that (getting a file from the command line) is that the desired order for repos such as ours is:

System -> User (happy to exclude with -Srepro) -> repo-master (this is missing) -> Project -> -Sdeps
and because "last one wins" etc for some combinations of aliases, you can't always specify "repo-master` via some sort of -Sdeps-file option.

seancorfield21:10:14

Although, if you could specify an arbitrary list of files to read in order via -Sdeps-file (and they didn't have to be called deps.edn) then we could use -Sdeps-file ../versions/master-deps.edn:project-deps.edn and not have an actual deps.edn file in each subproject in the repo.

borkdude21:10:46

-Sdeps-path might be a more appropriate name and I'd be happy to support it in deps.clj (and change the name). It already does this now but for just one file

borkdude21:10:10

-Sdeps-overrides could also work as a name to indicate that it doesn't read any other files than these

borkdude21:10:55

but I'm relatively sure all of this can be done using scripting as well: merge the configs yourself, provide it via -Sdeps and move the local deps.edn aside (hacky)

borkdude21:10:22

also I'm relatively sure these things are in some row/column in an excel sheet somewhere on Alex's computer :)

seancorfield21:10:46

Our deps files are pretty big -- too big to go on the command line I think as merged data -- and, as you say, that's a hacky approach.

seancorfield21:10:11

(also, we're not about to switch from the official Clojure CLI to a third party tool at this point, esp when we would need multiple different binaries for different parts of our dev/test/staging pipeline, whereas we can have just one CLI install)

seancorfield21:10:20

And, yeah, I've already brought this up with Alex (which was how I learned that it's similar to something the Datomic Ions users have asked for) so, yeah, I'll bet it's already in a spreadsheet. I don't know what I'd prefer in terms of a solution. As long as it addresses the deps ordering shown above and is officially supported, I'll be happy with whatever Cognitect/Nubank think is the best approach 🙂

borkdude21:10:38

@seancorfield small details: deps.clj also runs using clojure / JVM as a dependency but yeah I don't expect you to switch - I was just thinking out loud about this option. I'm blathering too much.

seancorfield22:10:02

Ah, true, I'd forgotten there was a library version as well...

borkdude22:10:52

I just pushed 0.0.10 - there was something with the clojars deploy token, missed that. But now it works:

/tmp $ clojure -Sdeps '{:deps {borkdude/deps.clj {:mvn/version "0.0.10"}}}' -M -m borkdude.deps -Sdeps-file "$HOME/git/clojurescript/deps.edn" -Spath
src/main/clojure:src/main/cljs:resources:/Users/borkdude/.m2/repository/com/cognitect/transit-clj/0.8.309/transit-clj-0.8.309.jar:/Users/borkdude/.m2/repository/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar:...

borkdude22:10:37

(so I'm in /tmp, but I'm using deps.edn from clojurescript in $HOME/git/clojurescript)

alexmiller23:10:16

Just putting this here for now, but there is a new prerelease clj available - 1.10.1.708. Has a small fix to -X error handling and more importantly an update on how dep expansion works when multiple transitive versions of the same dep are in the tree to fix a case where a dep can be incorrectly omitted.

alexmiller15:10:55

anyone tried this?

seancorfield16:10:21

I'm about to update things here...

seancorfield18:10:55

Building our uber JARs for our 14 processes seems to produce near-identical results between .697 and .708 -- there are variations from run-to-run based (I assume) on differing timestamps etc but nothing more than 150 bytes on JAR files that range from 30MB to 70MB, so that all looks reasonably sane.

seancorfield18:10:56

(if dependencies were missing we'd have known about it by now on .697 and 150 bytes here and there isn't likely to indicate an entire dependency coming or going)

alexmiller18:10:12

yeah, shouldn't be anything different other than the set of jars selected

alexmiller23:10:12

The circumstances for the latter case are quite complicated so it's a little challenging for me to categorically say whether your deps are affected (they probably aren't), but if you were were interested in trying it, would be happy to have any feedback before I make it stable.