Fork me on GitHub
#tools-deps
<
2018-10-11
>
seancorfield01:10:29

I needed very simple uberjar functionality with clojure.main as the Main-Class in the manifest. I forked and updated depstar to provide that https://github.com/seancorfield/depstar/commit/b49f15b0fc341f3004cb1bbf037334e3acbcb900 (a PR is in to @ghadi but my last PR to update the readme to explain the lack of manifest hasn't garnered a response yet so...).

seancorfield01:10:06

;; uberjar building:
  ;; - see  for a minimalist approach:
  ;; - uses my fork that adds manifest support
  ;;   - clj -A:depstar result.jar -m
  ;;     - sets clojure.main-main as the entry point
  ;;   - clj -A:depstar result.jar my.project
  ;;     - sets my.project/-main as the entry point
  :depstar {:extra-deps
            {com.healthfinch/depstar
             {:git/url ""
              :sha "a328136da59ea9b83483189451fb71aafd13bdb1"}}
            :main-opts ["-m" "hf.depstar.uberjar"]}

Michael Fiano06:10:05

As a beginner to tools.deps and Clojure in general, I just banged my head against the wall for a couple hours figuring this implementation detail out, because the documentation leaves off this edge case. I discovered something with alias loading I thought would be worthwhile to share here in any case: Assuming you have 2 aliases with :main-opts keys, regardless if other keys from other alias types are present in them or not: If -M is specified on the command line for both aliases, only the last alias is used as documented. If -A is specified on the command line for both aliases, only the last alias is used as per the documentation for -M. If -A is specified for one, and then -M is specified for the other, only the last one is used. If -M is specified for one, and then -A is specified for the other, still only the -M one is used, even though -A is later. It's that last rule that got me good. In short, you have to be very careful here.

😬 4
martinklepsch08:10:55

Anyone recommendations for a thing that will package my deps.edn lib into a jar and install it into the local repo?

martinklepsch08:10:23

Pack doesn't do regular jars, cambada doesn't install them into local maven, meyvn seems a little more involved

orestis08:10:26

I think the other one is depstar? No idea if it does what you want though.

mpenet08:10:27

yeah they all are a bit tricky to use, each with their own set of tradeoffs

mpenet08:10:08

personnaly I am half tempted to just rely on having clj installed and run apps with it, more or less what you do with cljdocs I think @martinklepsch right (bash script that downloads the archive, upacks it and runs)?

mpenet08:10:58

capsule based build seemed quite a good idea (one of the options on juxt.pack), but I am not sure where that project is heading

dominicm09:10:31

@mpenet happy to answer questions you might have!

martinklepsch09:10:42

I'm actually referring to regular jars (not uberjars)... e.g. when publishing a library

dominicm09:10:29

@martinklepsch fwiw, you could use pack with skinny jar mode and write a pom.xml. But you'd have to install that jar yourself (with maven I guess?)

mpenet11:10:40

@dominicm nothing special in mind. pack is imho the most complete solution so far. One minor missing piece would be the ability to specify a main on the one-jar option, but i can live without it

dominicm16:10:18

There should be an issue open for this. It is something I want to fix. I don't want to require AOT from users, so I want to patch onejar a little to support a default param to clojure.main.

orestis12:10:07

Would be nice if there was only one well supported solution for packaging. I stared with cambada but had to fork to apply some PRs

seancorfield14:10:15

@orestis sounds like me forking depstar to apply a PR to add a minimal manifest 😀

seancorfield14:10:31

@martinklepsch is there a reason you need a jar in your local repo? Can't you refer to that local library source via :local/root?

martinklepsch15:10:58

@seancorfield I can but I have some wierd interdependencies that aren't supported fully by Cursive/tools.deps

cfleming15:10:49

Right, transient :local/root deps are not well supported (by deps itself, and also Cursive - I’m fixing that right now)

orestis15:10:43

Even though I don’t use Clojure professionally yet, I’d chip in for an Arachne-like Kickstarter, but for a new build tool based on tools.deps and simple libraries.

orestis15:10:43

There’s so much duplication of effort these days - jar building, linters, formatters and so on.

orestis15:10:03

I have a dream for a build tool that would just provide an (overridable/extendable) set of library decisions and some glue code to invoke them from the command line.

Andreas Liljeqvist15:10:02

What you are describing is probably a program 😕

Andreas Liljeqvist15:10:15

But yeah, there is a lot of churn in the dev space right now

Andreas Liljeqvist15:10:41

Will probably be a while before it simmers down with some kind of best practices

orestis15:10:31

The whole point is to rally everyone around a single tool. I think this has multiplier benefits for the community and newcomers. See for example Elixir and their mix tool.

Andreas Liljeqvist15:10:18

There is certainly value for newcomers.

Andreas Liljeqvist15:10:56

Atm Lein is probably good for them

Andreas Liljeqvist15:10:45

But tools-deps is the first system that let me understand wtf was going on when starting up

orestis15:10:15

Yeah, I’m all for tools.deps for dependencies and classpath stuff.

seancorfield16:10:02

I've been recommending clj to newcomers -- with the caveat that most books/tutorials use lein. At least with clj, you can point newcomers at "official" documentation in a central place and you know the tooling is both simple and well-supported.

orestis16:10:12

As a relative newcomer, clj is just so much less cognitive overhead than Lein.

seancorfield15:10:25

For a long time, at work we've done no AOT and we build over uberjars to have clojure.main as an entry point. Our needs are very simple. We used to run a lot of stuff from source directly via lein and later boot but I think packaging as an uberjar for java -jar is simpler and safer (no need to fetch dependencies onto the target systems).

Andreas Liljeqvist15:10:12

+1 for building uberjars

seancorfield15:10:14

ClojureScript has had years of churn but seems to be settling down now. Maybe Clojure needs a bit of churn to shake things out and give us better, simpler tools too?

Andreas Liljeqvist15:10:23

tools-deps is a great step in that direction. But causes a bit of chaos in the transition

orestis15:10:06

To be clear, IMO tools-deps enables the creation of such a mostly-glue tool. Otherwise making a new build tool from scratch is a huge undertaking.

arrdem16:10:42

@martinklepsch the official story there seems to be write a pom.xml yourself, use the -Spom task to keep the dep lists in sync? (I don’t really understand what it does) and mvn install yourself.

dominicm16:10:05

Having attempted to write that build tool, I'm not happy with anything I've come up with. Build tooling is hard to get right, especially around tools like figwheel.

orestis16:10:17

Share notes perhaps? Any attempt to explore the space is very valuable.

dominicm16:10:17

Krei.alpha and kick.alpha are the two attempts by JUXT. We also did mach in the past which we used for alternative purposes.

dominicm16:10:08

@U064X3EF3 keeps mentioning that builds are programs, so I suspect that both of those attempts to be data driven are fatally flawed. I've been thinking of implement a paper by peyton jones for creating build systems a la carte, maybe that will unlock the pieces needed, but I'm not sure yet.

orestis16:10:06

My naive attempt so far is to just invoke some namespaces - either external or my own via aliases. My need is for more well rounded and robust libraries to do things like jars and so on.

dominicm16:10:36

Caching. Minimal builds. Early cut off. These are the general problems to solve. Cloud building is a nice to have in some contexts, but none I'm in.

orestis17:10:52

I’ll look at krei and kick, thanks.

orestis17:10:32

I don’t think that there is any way around the current slow startup of clojure, is it? Even with precompiled stuff, you will get at least a handful of seconds which is already too much. Hard stuff.

rickmoynihan17:10:46

you could always fire up a build-tooling repl once, that puts you in a namespace with functions like (build) (test) (do-everything)

zane17:10:40

@orestis Have you seen the results people are getting with GraalVM?

dominicm08:10:24

Something quite hard to build at a REPL is AOT-compiling namespaces. You have to start a new JVM to do that. You might be able to get away with something like a pod, but that's the best I have. Just as a result of how it decides to cache/skip namespaces.

lilactown16:10:03

especially since the UX is so dependent on perf. everyone wants things to be fast

lilactown16:10:07

or at least, fail fast

dominicm16:10:12

To be honest, I think you have to just make it a library that you can run from the repl. It has to happen around the Clojure command line tools, else there is too much friction.

dominicm16:10:02

Even shell scripts get in the way of tooling. Cursive won't work if you require funk.

seancorfield16:10:41

I really do like depstar's minimalism but it's frustrating that it omitted a minimal manifest -- and I'm not sure why it did that. Everything else in the build space is much more complex 😐

dominicm16:10:26

@seancorfield the easy answer is that generating manifests is out of scope. War is another format that needs a manifest to be written. Simpler to let users decide how to handle extra files in their classpath.

dominicm16:10:50

I don't mean to speak for the author, just a thought I've had working on pack.

rickmoynihan17:10:48

yes, not so minimal when you start having to do that…

seancorfield17:10:55

Adding a manifest with just a main class is only a few lines of Clojure when you're building a JAR tho'... 🙂

dominicm17:10:24

But it conflicts with any additional things a user may want to insert into a manifest themselves. Users can also generate a manifest in a few lines of code. In most cases of can just be hardcoded by the user.

seancorfield18:10:34

Right, which is why it's important that adding the minimal manifest is optional. depstar specifically excludes META-INF/*.MF files from the classpath so the resulting JAR is guaranteed to have no manifest. My fork allows users to conditionally add a minimal manifest that specifies Main-Class (only). That seems to be such a common need -- for java -jar ... -- that baking it in as an option seems worthwhile.

seancorfield18:10:31

The code is based on Boot's create-manifest so it could also take a hash map of key/value pairs to add as extra manifest attributes, but that's not exposed to the command-line in my fork (yet).

dominicm18:10:35

I see. That seems quite wrong. It also removes license files which is not allowed under some licenses like Apache

dominicm18:10:22

Fun fact: run manifest doesn't capture the full capabilities of manifests!

seancorfield18:10:37

Yeah, I noticed that and thought it was weird. I don't know why it does that (and Ghadi hasn't responded to any of my questions about it yet).

seancorfield18:10:47

We're not redistributing our JARs so the license issue isn't a problem -- and depstar filters out a lot of other stuff too so it's pretty brutal in that respect.

arrdem18:10:29

Is there an algorithmic reason beyond wanting to track the path for every expanded dep that expand-deps couldn’t group deps into resolution batches by manifest type? The Maven implementation used to read manifests seems designed for reading many manifests in one go, and less so for reading a single manifest at a time.

arrdem21:10:11

The hard thing is proving that letting Maven potentially solve range constraints instead of using deps’ behavior for everything can’t produce a different final dependency solution 😕

Alex Miller (Clojure team)22:10:26

Maven will produce a different solution, and we don’t want it

arrdem23:10:47

Ah. Yeah if you have a non-Maven pinning (or even just winning coordinate) for a dep, and Maven resolves something else, you can’t really back the transitive consequences of that out of Maven’s solution so you loose minimality for sure which can have other consequences.

Michael Fiano19:10:35

Regarding my message above about aliases, does anyone think this deserves an issue opened to at the very least document this edge case?

arrdem19:10:20

The order independence of -M winning when -A and -M is present is probably worth documenting.

arrdem19:10:25

So yes IMO.

dominicm19:10:30

I would prefer to see it fixed, but documentation would be an improvement over the current situation.

Michael Fiano19:10:30

Personally I'd like to see the behavior change. The reason why is this:

Michael Fiano19:10:41

: I have a :dev alias that is only ever project-local. I use it for example to require my namespace when starting up an nrepl in cider. This requires a :main-opts key, making it of the -M type, but as per the documentation, only the last -M type is ever used. I also want to compose this with a debugging alias to say depend on and require spyscope with another alias. Currently this is impossible without putting that in the :dev alias and thus copying it to be project-local instead of a global alias.

Michael Fiano19:10:00

Unless there is some other way to compose a global main-opts with a project local one that I do not know of

Michael Fiano19:10:04

Maybe I'm just asking too much from this too new tool?

seancorfield19:10:53

FWIW, I only ever use -A and I split aliases out so I run into this issue (downside: you need to specify multiple aliases for some things).

Michael Fiano19:10:25

Perhaps @U064X3EF3 can weigh in here as I see he is the project lead. When i get home later I'll open an issue if only to document this edge case so other newcomers don't bang their heads like I did.

andy.fingerhut19:10:16

It is better to ask about whether it can change when things are still alpha, than later. It is really up to folks like Alex, Rich, and Stu Halloway on how they choose to make the tool work, so don't be surprised with the answer "doc enhancement, please"

seancorfield19:10:22

My sense is that it is "by design" that aliases are combined in order, left-to-right and that -A:foo is just shorthand for -R:foo -C:foo -M:foo ... -- so the behavior you observe is intentional and likely considered to be already well-documented.

seancorfield19:10:58

If you specify your project-local alias with -R and/or -C instead of -A then you'll get the behavior you want.

Michael Fiano19:10:03

The confusing bit is that -M is documented as only the last one being used, but if -A is indeed shorthand for that, an -A after an -M needs to have a note mentioning that only the last explicit -M is applied, and -A is ignored.

seancorfield19:10:47

The -A is not ignored -- the aliases are all combined left-to-right, but with :main-opts it's always last one wins.

Michael Fiano19:10:58

But that is not the case.

Michael Fiano19:10:29

-M:foo -A:bar, when both foo and bar have :main-opts keys, then foo wins.

seancorfield19:10:21

Ah, gotcha. Here's why:

val="$(join '' ${resolve_aliases[@]})|$(join '' ${classpath_aliases[@]})|$(join '' ${all_aliases[@]})|$(join '' ${jvm_aliases[@]})|$(join '' ${main_aliases[@]})|$deps_data"

seancorfield19:10:21

So -M is always applied last. OK, yes, that should be documented. Do you want to create a JIRA ticket for that (against the CLJ JIRA project I believe)?

seancorfield19:10:57

I'm happy to create the ticket if you want me to.

Michael Fiano19:10:05

I very much want to so others don't run into the headaches I did. I won't be able to do that until tonight or tomorrow though when I'm at my dev machine, unless you don't mind.

seancorfield19:10:22

I'll go ahead and create it... against the website since it's really an error in the Reference page for the CLI

Alex Miller (Clojure team)22:10:32

This is a bug, not a doc issue. Order matters, should be respected.

seancorfield23:10:13

OK, good to know. So the shell script would need to expand each -A as it encountered it, adding the aliases as if all four were really specified explicitly?

seancorfield23:10:52

Do you want a JIRA issue (and maybe a patch) against CLJ for that instead?

Alex Miller (Clojure team)23:10:01

Jira yes, patch if you like. I suspect it affects the scripts too so may require some coordination

seancorfield23:10:28

Ugh! Yes, fixing the clojure script would solve one side of things -- that's easy -- but tools.deps also needs to be fixed too. OK. JIRA for now. I'll look at a patch if I get enough free time but it's a bigger change than I had initially thought.

Michael Fiano01:10:16

Thank you everyone.