Fork me on GitHub
#tools-deps
<
2020-03-25
>
Alex Miller (Clojure team)01:03:18

tools.deps is fundamentally about making classpaths, and incidentally about transitive dependency resolution

Alex Miller (Clojure team)01:03:03

I would not call it a package manager per se

đź‘Ť 8
Alex Miller (Clojure team)01:03:21

clj is about running programs

rickmoynihan10:03:23

I’ve been meaning to speak about this issue for a while; not sure if others have run into it or not. I have a private library in a private github repo; that we’re requiring via tools.deps. Every so often, in circumstances I don’t fully understand, but it usually appears to be when I do things like change branches (and I suspect the sha in that dep has changed) I get the following exception:

rickmoynihan10:03:13

When it occurs I usually spot that my ssh-agent is somehow no longer running; so restart that but then get:

rickmoynihan10:03:13

Then clearing the lock file fixes it

Alex Miller (Clojure team)12:03:27

We’ve had people report this but I’ve never been able to repro it

rickmoynihan14:03:20

It happens to me quite often

rickmoynihan14:03:41

there are a few other things that might be relevant: 1. some-private-lib is actually included in our deps.edn multiple times; with different :deps/root s 2. In dev I start a bunch of services with overmind a Procfile process manager. It boots them in parallel. One of those services is my clj repl; another is shadow-cljs that references the same deps.edn; so when resolving deps two processes are occaisionally concurrently resolving/fetching deps in parallel.

rickmoynihan14:03:52

I suspect 2 might be problematic… so sometimes try and resolve the deps first; then run overmind… though I do at times forget — it usually seems to work without any apparent problems, though it does seem a bit dangerous on my part.

Eamonn Sullivan14:03:47

I saw the first error just the other day. I had two small projects, one using a github ref using one hash (that checked out the repo fine) and then moved to another with a newer hash (but the same repo) and saw the above error when trying to build. I went into the directory and deleted the repo and tried again and it worked. Is that supposed to work?

rickmoynihan14:03:16

I think I’ve seen it a similar scenario to that too

rickmoynihan14:03:38

but this one is definitely within a single project — though changing branches; with a changing sha on the dep which probably amounts to the same thing

Alex Miller (Clojure team)14:03:54

with the newer clj's, resolution happens in parallel, so even in a single project, you could encounter that

Alex Miller (Clojure team)14:03:32

you can turn that off with -Sthreads 1

rickmoynihan14:03:11

Ok but this is in a single project; but with two concurrent processes potentially resolving the same deps with tools.deps. Is that safe if the processes aren’t coordinated?

Alex Miller (Clojure team)12:03:21

The first error that is

Alex Whitt13:03:49

Hi, I love the ability of tools-deps to target git repos at a certain hash. I've operated under the assumption that the hostname of the repo will stay the same forever for private repo dependencies. But that's proven not always to be the case. If a repo moves to a new hostname, it breaks every historical project state that referenced it. Yocto has a feature called a "repo manifest" that allows the repository configuration to be a lot more fluid, and effectively solves this problem. Does tools-deps have an equivalent option?

Alex Miller (Clojure team)13:03:09

what is the repo manifest?

Alex Miller (Clojure team)13:03:21

got a link or something?

Alex Whitt13:03:38

I suppose there are workarounds for the issue at the OS-level, but this would be a pretty slick feature.

dominicm14:03:03

@alex.joseph.whitt is this like having a proxy or alternative source for a repo? I'm struggling to understand.

Alex Whitt14:03:22

I'm not a Yocto user myself, but I believe the idea is that you can have a locally-defined manifest of repo sources. For each dependency, Yocto will look through them all in order and try to resolve the dependency in each. That solves the problem of trying to build an old version of your project after your git server hostname changes. The problem is really only sticky when you have multiple internal Clojure dependencies that all point to the same hostname. You have to go through each dependency, branch off of the contemporary commit, update the hostname, commit, then update the hashes on projects that pull in that dependency. Kind of hellish unless you redirect the old hostname at the DNS level or something.

Alex Miller (Clojure team)14:03:37

I mean, you are defining a lib and a coordinate (with git config) in your deps.edn. You can change that coordinate/git config but keep the lib name to point it elsewhere.

Alex Miller (Clojure team)14:03:31

I guess there could be a layer of indirection added in front of the git url

Alex Miller (Clojure team)14:03:41

needs a lot more thought

dominicm14:03:04

Another use case for this: the bank I work at has a github mirror which is both faster and allows clones from machines that otherwise can't. This would be useful for us.