Fork me on GitHub
#tools-deps
<
2020-01-29
>
dharrigan04:01:08

Happy to volunteer too!

๐Ÿ‘ 1
alexmiller04:01:50

the interesting bit is that parallelizing the jar downloads only got me so far because with Clojure artifacts, the pom is of significant size compared to the jar files and we end up downloading a lot more metadata during version selection

alexmiller04:01:02

so, working through parallelizing the metadata download too, which is a lot harder to work into the algorithm

alexmiller04:01:37

but, still seems doable

dominicm07:01:26

Very cool. Does maven do parallel at all?

alexmiller07:01:01

I have parallel metadata during expansion working now

๐ŸŽ‰ 3
alexmiller07:01:42

the benefits vary a lot depending on how much metadata/how much data/how much parallelism/how bushy the dep tree is, etc

alexmiller07:01:10

for something like the luminus template, for lein to download the full dep tree is ~43 sec

alexmiller07:01:19

clj before was 73 sec and is now 10 sec

๐Ÿ‘ 4
โค๏ธ 1
alexmiller07:01:52

that's kind of an extreme (lots of jars, huge fanout, lots of java, so big jars) etc

alexmiller07:01:30

but 20-60% improvement is a common range

alexmiller07:01:19

most of the time of course you're not really seeing those full numbers because you already have some or most of the libs downloaded in the cache

p-himik09:01:07

It seems that "RELEASE" and "LATEST" are not available in Maven 3, but tools.deps seem to support them. However, I cannot find any mention of these aliases in the documentation. What's the deal with them? Can people rely on them? Can developers safely recommend using them for e.g. such things as tools in the user's deps.edn? Also, I know that "RELEASE" is susceptible to cache issues - if no new dependency is needed to be retrieved during dependency resolution, the latest release available locally will be used (i.e., no check for a new release will be made) or something like that. Is "LATEST" also susceptible to such issues? Or does it reliably make a check for a new version each time the dependency resolution is done?

alexmiller13:01:26

They are not supported and break the cache checking

alexmiller13:01:21

That said, they do actually work and can be useful for one off calls in something like -Sdeps

alexmiller13:01:08

I recommend using real versions instead

p-himik13:01:41

@thheller ^ FYI Yeah, I use "RELEASE" only for clj-kondo, tools.deps.graph, etc. Something that runs only to facilitate the development.

๐Ÿ‘ 1
alexmiller15:01:41

note that if you use RELEASE for tools, you generally won't get new versions of the tools after you first run them, unless you -Sforce (because the hashed key is just seeing "RELEASE" every time)

seancorfield17:01:58

Good to know. Explains some of the slightly odd behavior I thought I was seeing from time to time since my dot-clojure deps.edn uses RELEASE a lot.

seancorfield17:01:39

clj-new has moved away from that and tracks explicit versions now.

๐Ÿ‘ 1
seancorfield17:01:32

It would be nice if there was some way in deps.edn to flag a dependency that should be rechecked periodically rather than cached indefinitely.

alexmiller17:01:18

The issue is that the cache check is done in bash to avoid jvm load so the hash is purely text based

alexmiller17:01:51

So there is no easy way to do what youโ€™re suggesting without a lot of trade offs

seancorfield17:01:38

Yeah, I think it would need an additional file in .cpcache for "dependencies that should be rechecked" which is probably more complexity than anyone would want for the occasional use of this feature.

seancorfield17:01:13

Is it worth me spending time thinking about this problem or would such a proposal not be considered anyway?

alexmiller18:01:44

Way down my list

seancorfield18:01:27

I'll take that as a "no" then ๐Ÿ™‚

alexmiller15:01:52

probably you're changing your local deps.edn or clj install frequently enough that you are periodically re-caching and don't notice this

borkdude15:01:48

it would be cool if tools.deps could resolve RELEASE in addition to tags maybe?

borkdude15:01:12

as in replace RELEASE with the most recent version in your deps.edn file

alexmiller15:01:53

I'm not planning on adding any more mucking with deps.edn

alexmiller15:01:14

but does not need to be in tools.deps - this itself could be a tool

alexmiller15:01:39

I'm not sure that's actually the best solution though

dharrigan15:01:36

https://github.com/Olical/depot does something similar, although I do not know if it processes "RELEASE" versions, but it updates the deps.edn in place definitely for deps that have versions.

dharrigan15:01:41

Maybe it does Besides SNAPSHOT versions --resolve-virtual will also handle the special version strings "RELEASE" and "LATEST"

borkdude15:01:40

This script uses RELEASE to print the most recent versions of libraries that are newer than you current ones: https://github.com/borkdude/babashka/#list-outdated-maven-dependencies A bit of a hacky light version of depot (thanks seancorfield)

robert-stuttaford16:01:53

can i remove a dependency in an override?

alexmiller16:01:53

iirc, no, but this has been on my radar for a while

robert-stuttaford16:01:23

I have datomic-pro overriding my tools.reader dep

robert-stuttaford16:01:34

looks like i'll have to use a cljs alias

thheller16:01:09

not exactly overriding. it contains a AOT compiled version of tools.reader which clojure will load instead of the actual tools.reader version on the classpath

alexmiller16:01:18

that's been fixed in datomic-pro, but not sure if that's out yet

๐Ÿ‘ 1
alexmiller16:01:46

yeah that fix is in 0.9.6024

robert-stuttaford16:01:00

thanks, @alexmiller!

rickmoynihan16:01:33

@alexmiller: I vaguely remember you once mentioning a tools.deps top level keyword for tooling to use that wouldnโ€™t pull in the main :deps? I canโ€™t see any mention of it, so Iโ€™m guessing itโ€™s not implemented yet?

borkdude16:01:26

you actually can remove a dependency with the classpath overrides

alexmiller16:01:35

if you are in an alias, you can use :deps to replace instead of :extra-deps to add

rickmoynihan16:01:46

ahh that was it, thanks.

borkdude16:01:55

^ setting the classpath override to nil will remove it from the classpath

borkdude16:01:28

only works in an alias I think

thheller16:01:35

or you could have a server/clj alias so cljs alias doesn't inherit all the other deps as well

robert-stuttaford16:01:39

@borkdude's suggestion worked! :cljs {:classpath-overrides {com.datomic/datomic-pro nil}}