Fork me on GitHub
#tools-deps
<
2020-01-29
>
dharrigan04:01:08

Happy to volunteer too!

👍 4
Alex Miller (Clojure team)04:01:50

the interesting bit is that parallelizing the jar downloads only got me so far because with Clojure artifacts, the pom is of significant size compared to the jar files and we end up downloading a lot more metadata during version selection

Alex Miller (Clojure team)04:01:02

so, working through parallelizing the metadata download too, which is a lot harder to work into the algorithm

Alex Miller (Clojure team)04:01:37

but, still seems doable

dominicm07:01:26

Very cool. Does maven do parallel at all?

Alex Miller (Clojure team)07:01:01

I have parallel metadata during expansion working now

🎉 12
Alex Miller (Clojure team)07:01:42

the benefits vary a lot depending on how much metadata/how much data/how much parallelism/how bushy the dep tree is, etc

Alex Miller (Clojure team)07:01:10

for something like the luminus template, for lein to download the full dep tree is ~43 sec

Alex Miller (Clojure team)07:01:19

clj before was 73 sec and is now 10 sec

👍 16
❤️ 4
Alex Miller (Clojure team)07:01:52

that's kind of an extreme (lots of jars, huge fanout, lots of java, so big jars) etc

Alex Miller (Clojure team)07:01:30

but 20-60% improvement is a common range

Alex Miller (Clojure team)07:01:19

most of the time of course you're not really seeing those full numbers because you already have some or most of the libs downloaded in the cache

p-himik09:01:07

It seems that "RELEASE" and "LATEST" are not available in Maven 3, but tools.deps seem to support them. However, I cannot find any mention of these aliases in the documentation. What's the deal with them? Can people rely on them? Can developers safely recommend using them for e.g. such things as tools in the user's deps.edn? Also, I know that "RELEASE" is susceptible to cache issues - if no new dependency is needed to be retrieved during dependency resolution, the latest release available locally will be used (i.e., no check for a new release will be made) or something like that. Is "LATEST" also susceptible to such issues? Or does it reliably make a check for a new version each time the dependency resolution is done?

Alex Miller (Clojure team)13:01:26

They are not supported and break the cache checking

Alex Miller (Clojure team)13:01:21

That said, they do actually work and can be useful for one off calls in something like -Sdeps

Alex Miller (Clojure team)13:01:08

I recommend using real versions instead

p-himik13:01:41

@thheller ^ FYI Yeah, I use "RELEASE" only for clj-kondo, tools.deps.graph, etc. Something that runs only to facilitate the development.

👍 4
Alex Miller (Clojure team)15:01:41

note that if you use RELEASE for tools, you generally won't get new versions of the tools after you first run them, unless you -Sforce (because the hashed key is just seeing "RELEASE" every time)

seancorfield17:01:58

Good to know. Explains some of the slightly odd behavior I thought I was seeing from time to time since my dot-clojure deps.edn uses RELEASE a lot.

seancorfield17:01:39

clj-new has moved away from that and tracks explicit versions now.

👍 4
seancorfield17:01:32

It would be nice if there was some way in deps.edn to flag a dependency that should be rechecked periodically rather than cached indefinitely.

Alex Miller (Clojure team)17:01:18

The issue is that the cache check is done in bash to avoid jvm load so the hash is purely text based

Alex Miller (Clojure team)17:01:51

So there is no easy way to do what you’re suggesting without a lot of trade offs

seancorfield17:01:38

Yeah, I think it would need an additional file in .cpcache for "dependencies that should be rechecked" which is probably more complexity than anyone would want for the occasional use of this feature.

seancorfield17:01:13

Is it worth me spending time thinking about this problem or would such a proposal not be considered anyway?

seancorfield18:01:27

I'll take that as a "no" then 🙂

Alex Miller (Clojure team)15:01:52

probably you're changing your local deps.edn or clj install frequently enough that you are periodically re-caching and don't notice this

borkdude15:01:48

it would be cool if tools.deps could resolve RELEASE in addition to tags maybe?

borkdude15:01:12

as in replace RELEASE with the most recent version in your deps.edn file

Alex Miller (Clojure team)15:01:53

I'm not planning on adding any more mucking with deps.edn

Alex Miller (Clojure team)15:01:14

but does not need to be in tools.deps - this itself could be a tool

Alex Miller (Clojure team)15:01:39

I'm not sure that's actually the best solution though

dharrigan15:01:36

https://github.com/Olical/depot does something similar, although I do not know if it processes "RELEASE" versions, but it updates the deps.edn in place definitely for deps that have versions.

dharrigan15:01:41

Maybe it does Besides SNAPSHOT versions --resolve-virtual will also handle the special version strings "RELEASE" and "LATEST"

borkdude15:01:40

This script uses RELEASE to print the most recent versions of libraries that are newer than you current ones: https://github.com/borkdude/babashka/#list-outdated-maven-dependencies A bit of a hacky light version of depot (thanks seancorfield)

robert-stuttaford16:01:53

can i remove a dependency in an override?

Alex Miller (Clojure team)16:01:53

iirc, no, but this has been on my radar for a while

robert-stuttaford16:01:23

I have datomic-pro overriding my tools.reader dep

robert-stuttaford16:01:34

looks like i'll have to use a cljs alias

thheller16:01:09

not exactly overriding. it contains a AOT compiled version of tools.reader which clojure will load instead of the actual tools.reader version on the classpath

Alex Miller (Clojure team)16:01:18

that's been fixed in datomic-pro, but not sure if that's out yet

👍 4
Alex Miller (Clojure team)16:01:46

yeah that fix is in 0.9.6024

rickmoynihan16:01:33

@alexmiller: I vaguely remember you once mentioning a tools.deps top level keyword for tooling to use that wouldn’t pull in the main :deps? I can’t see any mention of it, so I’m guessing it’s not implemented yet?

borkdude16:01:26

you actually can remove a dependency with the classpath overrides

Alex Miller (Clojure team)16:01:35

if you are in an alias, you can use :deps to replace instead of :extra-deps to add

rickmoynihan16:01:46

ahh that was it, thanks.

borkdude16:01:55

^ setting the classpath override to nil will remove it from the classpath

borkdude16:01:28

only works in an alias I think

thheller16:01:35

or you could have a server/clj alias so cljs alias doesn't inherit all the other deps as well

robert-stuttaford16:01:39

@borkdude's suggestion worked! :cljs {:classpath-overrides {com.datomic/datomic-pro nil}}