This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-08-09
Channels
- # announcements (16)
- # beginners (86)
- # calva (4)
- # cider (17)
- # circleci (1)
- # clj-kondo (4)
- # cljs-dev (12)
- # cljsrn (4)
- # clojure (82)
- # clojure-europe (2)
- # clojure-houston (4)
- # clojure-italy (5)
- # clojure-nl (7)
- # clojure-spec (49)
- # clojure-uk (19)
- # clojurescript (76)
- # core-async (7)
- # cursive (1)
- # data-science (4)
- # datomic (5)
- # figwheel (1)
- # fulcro (10)
- # graalvm (15)
- # jobs (1)
- # juxt (6)
- # kaocha (2)
- # leiningen (5)
- # random (2)
- # shadow-cljs (25)
- # sql (5)
- # tools-deps (113)
- # vim (3)
- # yada (14)
The piece that is more interesting to me is shortening the time it takes to get a classpath. We have some services where it takes nearly a minute to get a classpath.
are you i/o bound (and if so, why?) or is the tree huge? or it something in resolution (cpu bound)?
a minute seems super long so would like to learn more about why
also, curious what version of clj you're on? Just reading the code, it seems like the exclusions should get canonicalized in this scenario, at least in current code
oh, I see why
Are there transitive dependency reachability checks done using exponential time algorithms? 🙂
Probably a completely irrelevant remark -- I just recall finding and fixing such a thing in tools.namespace years back.
it's designed to be iterative and should be a single pass over the tree
but problems like missing a cycle could cause issues
@kenny I found and fixed the bug with exclusion canonicalization, really a problem with exclusions in any deps.edn, not just transitive. very good catch!
not going to release it right now but will be in whatever the next release is
I looked at our deps and we certainly have a mix of bare and canonicalized exclusions so it'll be interesting to see what affect that has on us 🙂
my read of current release is that all bare exclusions are being ignored
Thanks. I'll try to qualify all of ours and see if anything falls out...
@alexmiller Thanks for the fix! > are you i/o bound (and if so, why?) or is the tree huge? or it something in resolution (cpu bound)? Just on a regular computer w/ ssd so not i/o bound. The tree is pretty big but I'd expect a tree this size with any enterprise product. Could be something in resolution that is slow, don't think it's cpu.
I mean i/o bound in talking to the network to download jars
not filesystem
like during that minute, could you grab periodic stack traces, either with ctrl-\ or with jstack, and look at the top of the stack?
so parallel downloads would not help you at all if you're not downloading anything
another debug thing to do is -Sdeps '{:aliases {:v {:verbose true}}}' -A:v
if you see pauses in there, that would be suspicious, but otherwise, if you just have a big trace, I'd be interested in seeing that, could dm it to me
not using which cache? classpath cache? m2 local repo? gitlibs cache?
so you're not actively clearing the m2 repo or anything
so really classpath cache
that's the only thing stale when you change deps.edn
and you're on latest clj?
there were some cycle detection issues that were fixed months ago
clj -Sverbose
for version
yeah, that's latest
well, I'd love to take a look
For starters, these messages have reappeared and they take a decent chunk of time:
Downloading: io/grpc/grpc-api/maven-metadata.xml from
Downloading: io/grpc/grpc-core/maven-metadata.xml from
Downloading: io/grpc/grpc-netty-shaded/maven-metadata.xml from
@seancorfield had noticed that a pom from one of my deps (com.google.cloud/google-cloud-monitoring "1.78.0") used a RELEASE version. He suggested explicitly specifying the deps mentioned there. I have done that and they still appear.yeah, that's actually an s3 wagon issue, upstream from tools.deps
He also mentioned it may have something to do with not correctly resolving deps from a parent pom.
I don't have enough info to debug this, would be useful to see deps.edn and the verbose trace above
Just this will do it:
{:deps {com.google.cloud/google-cloud-monitoring {:mvn/version "1.78.0"}}
:mvn/repos {"datomic-cloud" {:url ""}}}
this is starting to ring a bell
there's a loop in these maven deps iirc
Adding time
to the clj
calls show I drastically overestimated the time it takes:
real 0m23.962s
user 0m46.126s
sys 0m1.158s
This certainly feels like an eternity when needing to do that many times a day. A big portion of that time is the "Downloading: ..." thing. It would be a huge productivity boost to get that under 5s.that's an issue with the s3 wagon I think
without the datomic repo in the mix, it's about 5 seconds to build a classpath for that
That s3-wagon thing has always been a nightmare for me. I remember always hitting issues with it back when we used an s3 maven repo. Perhaps a good use case for aws-api? 🙂
with it, I see about 8-9 seconds
time clj -Spath -Sforce
yeah, that's the idea :)
any success stories of multi-module/mono-repo library setups with deps? have some working lein projects for that, but have now a deps project that needs to be split into parts.
@ikitommi We switched to a monorepo a month or so ago. All internal libraries are :local/root
. Makes working in the REPL great. There's a few kinks with our setup though:
- We use CircleCI for CI/CD. There's no support for monorepos with CircleCI. This leads to longer build times - every project runs through its test steps with every push. We recently switched to their new unlimited parallelism plan which has been quite helpful in getting CI time down.
- CI configuration was moved into a set of clojure files because it became far too tedious messing around with YAML with the number of projects we have. This does, unfortunately, mean that CI config needs to be manually generated with a command every time you change the CI clojure files.
- We have a small service diff library that detects when a particular service's code, deps.edn, or :local/root
deps have changed. That ensures a service won't get deployed on every push.
- We don't have a great way to have a common deps.edn across all projects. This would be quite useful for things like: global exclusions (when supported), overriding certain library versions, common aliases, etc. Ideally there'd be some way to just pass in N number of deps.edn files to clj
and have it merge those in. I use Cursive so Cursive would also need to have some way to select which deps.edn files to use.
- We don't have a good way to run commands across all projects or only in a certain project. For example, I'd like to be able to do something like: monorepo my-service uberjar
.
- Different libraries & services run tests with a different set of aliases. Every time we want to run the tests for a project, you need to go to the projects README (or check the CI config) and determine which aliases to use to run the tests. Either the aforementioned "command runner" or a way to combine aliases somehow would make this much better.
Overall, this workflow is far better than having individual repos and constantly needing to restart the REPL when working across projects.
Thanks @kenny! was hoping for the monorepo
kinda script, too lazy to start cooking up own tools right now. In my case, it's a library, going to be split into set of libraries, so would be easy to have same aliases for all. A sample repo would be super awesome.
We have a monorepo with maybe two dozen subprojects, and 90k lines of Clojure.
The key thing we did was to have a primary deps.edn
in a folder and point to that via CLJ_CONFIG
(so it "replaces" the user-level deps.edn
) and then each subproject has a deps.edn
.
We use :override-deps
in the primary file to "pin" versions of libs across the whole repo as needed, as well as provide all the common tooling via aliases.
The only "tooling" we've built on top of this is a small shell script that can execute multiple clojure
commands and knows how to navigate to subprojects when running series of commands.
Like @kenny we use :local/root
deps for cross-module deps -- and we have an everything
subproject that we can build the deps.edn
into from across the monorepo and that's where we usually start our REPL/REBL from.
Hmm that's a good idea! Cursive doesn't support CLJ_CONFIG
unfortunately. Creating a everything
project and starting a nrepl from the command line could solve that problem though! Generally it makes sense to have everything on the classpath while dev'ing.
build set:of:aliases subproject
is our shell script. But it can take multiple pairs of aliases/subprojects.
and if we need arguments, we can use [
]
to wrap them, so build uberjar api [ run ci-ftp api [email protected] ]
How do expose build
? Is it a script at the root of the repo? Do you have devs add to PATH?
We have a <repo>/build/bin
folder containing scripts. Devs can either add it to their path or just run the scripts directly.
I mostly work in the build
folder but I have build/bin/build
symlinked into my ~/bin
folder for convenience. Other stuff I run with ./bin/<script>
Between docker compose
and two git clone
commands, a dev can be set up "immediately" (assuming they have an OpenJDK8 installed).
If they need to work on our legacy apps, there's one more git clone
to run.
(one of those repos is for semi-static tooling, which is where we run docker compose
-- for Redis, Elastic Search, Percona/MySQL, and a custom search engine we use)
I’ve been using this trick to “bless” <repo>/bin
path additions: https://thoughtbot.com/blog/git-safe
Interesting little trick!
Nope. We use "../versions"
in our script.
CLJ_CONFIG=../versions clojure -A:defaults:<task> <args>
:defaults
pulls in all the overrides etc from versions/deps.edn
For example: if I have a build/bin/build and I run it from build/bin, I need to know to set CLJ_CONFIG to ../../versions. If I run it from build, I need to set CLJ_CONFIG to ../versions.
dirname $0
Then you can work relative to that.
(although we assume certain filesystem paths are the same on all dev/test/prod images so some of our scripts take advantage of that -- and devs just add a symlink to wherever they decided to put stuff)
Given you symlink build, dirname $0
will return a path to wherever the symlink is. How do you deal with that?
Yes, you can use readlink $0
to get the actual file location (it exits with a non-zero status if the argument is not a symlink).
Like I said above, we also assume certainly filesystem paths to make our lives easier 🙂
(partly because we have a shell script in the main repo that a new dev can download and it does most of the env setup for them, including git clone
ing repos to specific places and setting up symlinks and handling the initial Mac/Linux differences)
But there are plenty of ways to skin that particular kitty.
What is the difference between:
• (resolve-deps {:paths [p1 p2] ...})
,
• (make-classpath ... [p1 p2])
and
• (make-classpath .. nil {:extra-paths [p1 p2]})
All three seem to produce the same result, which is to add the local project's paths p1
and p2
to the classpath.
To further exemplify my question from yesterday, I'm looking for something on the lines of:
{:deps {some/dep {:git/url ... :paths [p1 p2]}}}
Is the example above feasible?
Those 3 produce the same result but are semantically different
What’s your goal?
I want to add a folder (for example 'test') from a dependency I loaded w/ tools.deps to my currently running project's classpath.
That folder is not declared on the main {:paths [...]}
clause on dependency's deps.edn for obvious reasons, it is not the mainline for that project.
However, as I'm trying to build something like a buildscript CLI interface for a group of projects, I want to dynamically load them using tools.deps and run their tests, or whatever else I might want to do with them, given that not only I can load their dependencies but also add folders (or other aliases from that deps.edn) into this buildscript CLI classpath.
There's a hacky way for me to work around that, which is to hijack the result of (result-deps)
through some (update-in (result-deps ...) [dep :paths] conj '/my/hand-crafted/path/p2')
before (make-classpath ...)
. That seems too hacky for me, but I can do that if tools.deps
doesn't want to explore further the project structure of a dependency.
If you’re calling tools.deps programmatically then you’re already in the machine - do whatever you want with the intermediate results
@ikitommi I added a build
script to our repo that is similar to the one @seancorfield described. Here's what I ended up with. Our repo is structured like with all projects under projects
and this script located at bin/build
.