Clojurians Log v2
babashka
Given a monorepo with many bb.edns within it, what would be an idiomatic way to resuse some defns across them? (e.g. put them somewhere at the monorepo root)
I'd be seeking something lightweight, no installation steps or versioning involved
Make a small lib with :local/root
I quickly tried it under ./bb.edn
{:tasks {:init (defn foo! [& cmd-args]
(prn 42))}}:deps {foo/foo {:local/root ".."}}foo! defn isn't available.:local/root must point at a deps.edn, not a bb.edn
all deps related stuff goes through deps.edn
Hmm, okay. That might be slightly verbose but can try
Should load-file work?
yes, load-file also works. or you can dynamically add to the classpath with (babashka.classpath/add-classpath)
Dear all - Ihave got an error on one of our internal machines - the error is pretty clear but I wonder if anybody has seen it and why it can happen...
'/root/.gitlibs/_repos/https/github.com/grzm/awyeah-api/info/exclude': File exists
Exception in thread "main" clojure.lang.ExceptionInfo: babashka.process.Process@19010461
at babashka.process$check.invokeStatic(process.cljc:111)
at babashka.process$shell.invokeStatic(process.cljc:677)
at babashka.impl.deps$add_deps$fn__28261.invoke(deps.clj:107)
at borkdude.deps$_main.invokeStatic(deps.clj:1071)
at borkdude.deps$_main.doInvoke(deps.clj:901)
at clojure.lang.RestFn.applyTo(RestFn.java:140)
at clojure.core$apply.invokeStatic(core.clj:667)(deps/add-deps (select-keys bb-edn [:deps :paths]))
to add the deps from a bb.edn dynamicallyI have a kind-of repro here but I'm not sure if it's the same error you're facing.
(def fut1 (future (babashka.deps/add-deps '{:deps {com.grzm/awyeah-api {:git/sha "e5513349a2fd8a980a62bbe0d45a0d55bfcea141"
:git/url ""}}})))
(def fut2 (future (babashka.deps/add-deps '{:deps {com.grzm/awyeah-api {:git/sha "9257dc0159640e46803d69210cae838d411f1789"
:git/url ""}}})))
[@fut1 @fut2] rm -rf /tmp/g1 && GITLIBS=/tmp/g1 bb /tmp/foo.clj$ rm -rf /tmp/g1 && GITLIBS=/tmp/g1 bb /tmp/foo.clj
Cloning:
Cloning:
Error building classpath. Unable to clone to /private/tmp/g1/_repos/https/github.com/grzm/awyeah-api:
fatal: destination path '/private/tmp/g1/_repos/https/github.com/grzm/awyeah-api' already exists and is not an empty directory.
Checking out: at e5513349a2fd8a980a62bbe0d45a0d55bfcea141 Same repro with JVM Clojure only:
(require '[clojure.repl.deps :as deps])
(def fut1 (future (deps/add-lib 'com.grzm/awyeah-api {:git/sha "e5513349a2fd8a980a62bbe0d45a0d55bfcea141"
:git/url ""})))
(def fut2 (future (deps/add-lib 'com.grzm/awyeah-api {:git/sha "9257dc0159640e46803d69210cae838d411f1789"
:git/url ""})))
[@fut1 @fut2] $ rm -rf /tmp/g1 && GITLIBS=/tmp/g1 clj <<< '(load-file "/tmp/foo.clj")'
Clojure 1.12.2
user=> Execution error (ExceptionInfo) at clojure.tools.deps.interop/invoke-tool (interop.clj:81).
Unable to clone /private/tmp/g1/_repos/https/github.com/grzm/awyeah-api
fatal: destination path '/private/tmp/g1/_repos/https/github.com/grzm/awyeah-api' already exists and is not an empty directory.the latest error seems the same but I am not doing anything in parallel here - as you can see I am only invoking the run.clj command - I suspect the parallel call happens internally to bb
Could that be the case?
going to try your repro
$ rm -rf /tmp/g1 && GITLIBS=/tmp/g1 bb subdir/another-dir/run.clj
Cloning:
Checking out: at e5513349a2fd8a980a62bbe0d45a0d55bfcea141
Downloading: com/cognitect/aws/api/0.8.762/api-0.8.762.pom from central
Downloading: com/cognitect/aws/s3/871.2.32.2/s3-871.2.32.2.pom from central
Downloading: com/cognitect/aws/endpoints/871.2.32.30/endpoints-871.2.32.30.pom from central
Downloading: tolitius/inquery/0.1.22/inquery-0.1.22.pom from clojars
Downloading: tick/tick/1.0/tick-1.0.pom from clojars
Downloading: fipp/fipp/0.6.29/fipp-0.6.29.pom from clojars
Downloading: tolitius/yang/0.1.50/yang-0.1.50.pom from clojars
Checking out: at 9257dc0159640e46803d69210cae838d411f1789
Downloading: com/cognitect/aws/api/0.8.762/api-0.8.762.jar from central
Downloading: com/cognitect/aws/endpoints/871.2.32.30/endpoints-871.2.32.30.jar from central
Downloading: com/cognitect/aws/s3/871.2.32.2/s3-871.2.32.2.jar from central
Downloading: fipp/fipp/0.6.29/fipp-0.6.29.jar from clojars
Downloading: tolitius/inquery/0.1.22/inquery-0.1.22.jar from clojars
Downloading: tick/tick/1.0/tick-1.0.jar from clojars
Downloading: tolitius/yang/0.1.50/yang-0.1.50.jar from clojars
Downloading pod org.babashka/postgresql (0.1.0)
Successfully installed pod org.babashka/postgresql (0.1.0)
---
/private/tmp/babashka-deps-repro/lib/lib-a
BEFORE add-dependencies
ℹ️ Available command ℹ️ $ rm -rf /tmp/g1 && GITLIBS=/tmp/g1 bb subdir/another-dir/run.clj
Cloning:
Checking out: at e5513349a2fd8a980a62bbe0d45a0d55bfcea141
Checking out: at 9257dc0159640e46803d69210cae838d411f1789
---
/private/tmp/babashka-deps-repro/lib/lib-a
BEFORE add-dependencies
ℹ️ Available command ℹ️ repeating this a few times gives the same results and no, bb doesn't fetch any deps in parallel
unless you do it yourself in a thread
can you repro it locally?
@richiardiandrea could it be that you are referring to awyeah-api with two different libnames (the key in :deps)?
something like this:
(require '[babashka.deps :as deps])
(deps/add-deps '{:deps {com.grzm/awyeah-api-1 {:git/sha "e5513349a2fd8a980a62bbe0d45a0d55bfcea141"
:git/url ""}
com.grzm/awyeah-api-2 {:git/sha "9257dc0159640e46803d69210cae838d411f1789"
:git/url ""}}}) checking
I can repro locally but only in my other repo - not in the one I sent you
check and the dep name was the same - I had :git/tag in there as well
perhaps you can make the repro more like your own repo?
yeah that's what I tried in the repo 😄
the dir structure is exactly the same
probably there is still something missing though - I am sure that if I find what triggers it we'll find the problem
for what is worth I am using v1.12.207
208 is out btw
I even removed the bb.edn from lib-b as per my project but that did not make any difference
in my project I see two concurrent Cloning...
wondering if bb is finding another bb.edn somewhere on the paths and launching itself twice but this is a wild wild guess
@borkdude I bumped to 208 and the problem went away!
Clojure tools not yet in expected location: /Users/.../.deps.clj/1.12.2.1565/ClojureTools/clojure-tools-1.12.2.1565.jar
Downloading to /Users/.../.deps.clj/1.12.2.1565/ClojureTools/clojure-tools.zip
Unzipping /Users/.../.deps.clj/1.12.2.1565/ClojureTools/clojure-tools.zip ...
Successfully installed clojure tools!
Cloning:
Downloading: org/clojure/clojure/1.12.2/clojure-1.12.2.pom from central
Checking out: at e5513349a2fd8a980a62bbe0d45a0d55bfcea141
Checking out: at 9257dc0159640e46803d69210cae838d411f1789
Downloading: org/clojure/clojure/1.12.2/clojure-1.12.2.jar from central Cloning anymoreww nice!
Now I am curious to see the release notes 😄
perhaps a change in tools.deps / deps.clj
probably best to go through the commits between 207 and 208
@richiardiandrea We can try this:
DEPS_CLJ_TOOLS_VERSION=1.12.1.1550 bb ..DEPS_CLJ_TOOLS_VERSION=1.12.1.1561 bb ...1.12.2.1565 and 1.12.2.1571in 208 the default version is 1.12.2.1565
hmm
have you perhaps messed with .gitlibs manually?
hmm no this is a machine I don't have control over and does not happen locally
ok I deleted my local gitlibs and now I see the error
wait - maybe it's on my side
I see it is trying to clone twice
my guess would be that tools.deps is trying to clone a git lib where the dir already exists or so
it might be the case that bb itself is download because it see it in bb.edn - then I add another one with deps/add-deps
but something is definitely going on
Cloning:
Checking out: at e5513349a2fd8a980a62bbe0d45a0d55bfcea141
Checking out: at 9257dc0159640e46803d69210cae838d411f1789 and it happens before the call to deps/add-deps
can you reproduce it?
trying
yeah so this happens when a library includes the same dependency with a different git/sha
I can push a repro in a second but I probably should match the git/sha 😄
interesting :)
this is my mess - but because it's interesting I am about to send you a repo 😄
normally tools.deps compares git/sha and selects the newest I believe, so yes, repro more than welcome
😄
I am using :local/root for libs - that's maybe why
could also be something suboptimal with babashka.add-deps, which doesn't care what's already on the classpath, it just adds more stuff to it
hmm strange - cannot reproduce anymore 😞
when I can repro I see two
Cloning:
Cloning: sorry I could not make a repro with this - it seems finicky and might depend on my repo dir structure...
I'll leave the repo here though. in case you want to try https://github.com/arichiardi/babashka-deps-repro
will try more tomorrow and please let me know if you want me to try a bb with more logging.
Could it be that you are using parallel babashka tasks? Perhaps tools deps isn’t prepared for concurrent downloads