Fork me on GitHub
#tools-deps
<
2020-08-16
>
borkdude12:08:17

@seancorfield Would it be possible to exclude clojure and clojure.spec.alpha sources/AOT-ed assets from an uberjar? The use case would be for babashka which already has clojure built-in (spec not yet, but maybe coming). So an uberjar with only (non-clojure) source files and no AOT-ed stuff would be leaner. https://github.com/borkdude/babashka/issues/535#issuecomment-674518153 Since you're reading from a system property here https://github.com/seancorfield/depstar/blob/b7e9e4537e33bbf78fbdbc0ef56d9675cb494bc2/src/hf/depstar/uberjar.clj#L283 it might be a small tweak to override the classpath with a command line arg or file reference (java itself uses the @ convention for that).

borkdude12:08:09

E.g.:

clojure -A:depstar -m hf.depstar.uberjar MyProject.jar -v -m proj --classpath "$(clojure -Spath)"

borkdude12:08:57

Btw, I think depstar will be able to compile with GraalVM so maybe I can even include it in babashka itself to create babashka uberjars

💯 3
Alexis Vincent16:08:46

I’d like to distribute a deps.edn file of aliases for everyone to use in our company. But I want people to be able to have their own local ./clojure/deps.edn. Is there a way to specify a deps.edn file to merge in, either globally for all projects, or from a projects deps.edn?

mhcat17:08:28

Is it possible to add a list of jar files from a project sub-dir to the classpath without having to name them? I know about :local/root and I see this in the docs https://clojure.org/guides/deps_and_cli#local_jar but I have a lot of them and they aren't published anywhere. Do I just have to make up names for them and add them individually as :deps?

Alex Miller (Clojure team)17:08:33

It’s kind of a hack, but java supports on the classpath for this (literal path/ so not shell expanded). I think if you added a :paths entry for “path/*” that should work

rickmoynihan08:08:49

Interesting I was wanting to do something similar too, so I could deploy an app as a zip with a directory of jars in lib but start it with the clojure command line tooling; via a :local/root; with a different deps.edn for production, rather than the one we use in dev.

Alex Miller (Clojure team)13:08:08

the other crazy thing you can do is to reference jars in your manifest (this was part of the original applet support) with the Class-Path attribute but I think that has to be explicit jars. afaik that does not support the glob stuff

rickmoynihan14:08:34

yeah now you mention it I’ve seen manifests do this…. a long time since I’ve seen or written an applet tho! 😆

Alex Miller (Clojure team)14:08:06

also relevant, the manifest has a line wrapping limit and so other arcane rules so it's best to use the apis to make it rather than doing it by hand

rickmoynihan14:08:06

wow I had no idea… eitherway if I do this I think it’ll be via the :local/root method, largely motivated via the new -F -X stuff

mhcat17:08:58

thank you @alexmiller I will try that out

mhcat17:08:13

amazing, thank you Alex, it works

👍 3
borkdude19:08:50

@seancorfield Awesome: depstar works with GraalVM and can hence be included in babashka proper so it can make its own uberjars

borkdude19:08:36

$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj
Building uber jar: foo.jar
$ bb -jar foo.jar
:foo - failed: int?

seancorfield20:08:13

@borkdude Cool. As for your earlier question, depstar deliberately just includes everything on the current classpath, aside from itself.

ghadi20:08:28

That was a fundamental design decision I regret

ghadi20:08:13

In retrospect I’d use t.d.a as a library to calculate what to tar up

ghadi20:08:37

rather than subtracting depstar itself from the classpath

borkdude20:08:21

yep. but that can be easily changed. I've already done that in my fork

ghadi20:08:28

You using t.d.a to calculate instead?

seancorfield20:08:26

To be honest, using the classpath has made depstar a lot easier to use for people, and a lot more predictable.

borkdude20:08:06

@ghadi I just added a :classpath option that can be used to override

seancorfield20:08:00

Hmm, interesting approach. Since it's easy to get clojure to produce an arbitrary classpath from deps.edn aliases etc...

ghadi20:08:08

this is a similar thing to what I’m trying to avoid, vectors of strings

borkdude20:08:54

So the classpath given to babashka, produced by clojure is just passed down to depstar:

$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj
which results in a foo.jar that can be invoked with bb -jar foo.jar.

ghadi20:08:43

I understand. I’m just saying the shell is not leverage compared to a clojure library. One of the things I want to do is split a list of dependencies by their rate of change for smarter distribution to containers or Lambdas

borkdude20:08:01

You mean, make uberjars for things not changing often?

ghadi20:08:08

For containers, you don’t want every commit to result in a 60MB new uberjar, you want some structural sharing across commits

borkdude20:08:15

I guess one could use the -A or -T option for that, to make groups of deps and then push that classpath into depstar

ghadi20:08:57

Yeah I know you can hack it in shell, but why not program it directly in bb or clojure?

ghadi20:08:34

my hope with tools.build (not that I have anything to do with it currently) is that it encourages build tasks to compose better than the lowest common denominator of the CLI

borkdude20:08:54

bb will have an --uberjar option, but it can also have a bb -e '(babashka.uberjar/run {:classpath ...})-ish interface

borkdude20:08:17

can have them both

vlaaad20:08:36

Wait are tools.build out?

seancorfield20:08:13

depstar currently has zero dependencies, except Clojure itself, so it can't interfere with anyone's project dependencies -- that's worked very well in practice. I like @borkdude’s idea of just giving it an explicit classpath to build form 🙂

vlaaad20:08:40

Unless I want to have the whole build happen in a single vm, and it has to have other build-specific dependencies like data.xml for updating pom.xml. I use 2 vms for that now, but I'd rather not :)

seancorfield21:08:32

I'm not sure whether you're agreeing with depstar's approach or suggesting something different?

borkdude20:08:33

bb won't do AOT btw (maybe it could but I don't think it's worth it). for JVM uberjars (with AOT-ed classes), just use clojure / depstar

borkdude20:08:53

bb uberjars typically will contain sources and other assets, but no classes

borkdude20:08:46

I did have depstar compile with GraalVM as is and when I didn't remove the AOT-related code, it did compile, but the binary got somewhat bloated (+ 30MB, happens when it hits a specific part of Clojure). So maybe compile just works natively, since it's just going to transform .clj into classfiles, I haven't tried it.

borkdude20:08:04

I'll try it now... just for science

borkdude21:08:30

Interesting. Compilation of a class doesn't work in a GraalVM binary because the classpath inside the binary is not set correctly for clojure RT. I guess it won't work for that reason, although technically it could maybe work.

borkdude21:08:27

Going to try this:

(binding [*compile-path* (str tmp-c-dir)]
          (clojure.lang.RT/compile (str main-class ".clj")))

borkdude21:08:51

crap, it's private ;)

borkdude21:08:31

With a reflective hack it works. I see all kinds of AOT-ed sourced, even aot__init.class, but no aot.class. Maybe :gen-class isn't part of RT/compile? Eh.... enough for today.

borkdude21:08:01

So the AOT part could be useful when depstar had a binary CLI version, so to speak. I don't need it for bb, but it could be useful for fast build tooling in general.

borkdude21:08:58

To make this work clojure.lang.RT should also have a pluggable classpath resolver. It might work for Clojure files, but I wonder how it will work if code relies on .class files from the Java world. Does the Clojure compiler kind of treat .class files as data, without actually having to have them on the classpath per se, as long as you point it to the right file? I.e. no runtime need, just able to read and analyse statically?

borkdude22:08:12

I see, Compiler relies on reflection when dealing with interop, so there it will likely break, since the reflection API only works with classes that are part of the same runtime. OK, this was a nice tangent.