Clojurians Log v2

Yeah, it sounds far fetched because it's so nonsensical.

Still picking apart this problem.

This might be due to one task not having deps and another one having deps?

I looked for that too, they have exactly the same :requires

The file is written by deps.clj where it wants to unpack the tools jar: https://github.com/borkdude/deps.clj/blob/ad71e665843f133c164ffd73d3444f8b81580eea/src/borkdude/deps.clj#L415 you can set this location manually with DEPS_CLJ_TOOLS_DIR https://github.com/borkdude/deps.clj/tree/master?tab=readme-ov-file#deps_clj_tools_dir

is there :extra-deps or perhaps babashka.deps/add-deps somewhere

If the project is open source, I'd have a look

Thank you, it's not 😞 There's :extra-deps for the :test alias (which runs without issue for the build I'm looking at)

I did see the thing about manually setting DEPS_CLJ_TOOLS_DIR in the README, but I thought it not relevant since other tasks run without issue?

Thanks for the links, I was in doubt about where to find TRANSACTION_START in source.

My coworker suspects it's to do with Jenkins nodes swapping work mid-process, or something to do with them sharing the file system, or something to that effect.

In that case set the env variable to some stable disk

Indeed 🙂 Need to know more about the Jenkins environment.

If you change Jenkins agent mid-build you will most likely lose any files you had downloaded.

These days Docker files often change the user towards the end of their setup as Dockerfiles start out as root and you want to run your app as a less privileged user. Changing the user would likely change the meaning of ~.

That shouldn't be intermittent though 🤔

Is there an easy way to determine where Clojure Tools was put by an initial call to bb prepare so that I can se DEPS_CLJ_TOOLS_DIR to that path?

you should see logs of the tools jar being downloaded if it's not there yet

This is the logic: https://github.com/borkdude/deps.clj/blob/ad71e665843f133c164ffd73d3444f8b81580eea/src/borkdude/deps.clj#L770

if you don't set anything, it will use the HOME environment directory

so $HOME/.deps.clj

on my system that is:

~/.deps.clj/1.12.1.1550/ClojureTools/clojure-tools-1.12.1.1550.jar

Yes, I see. The thing is, I'm trying to call bb prepare in a RUN command in a dockerfile, and it seems like that ends up somewhere else (or with a different version) than when I call bb prepare while the container is actually running at some later point. So I'm thinking I have to extract the path during the initial download, set DEPS_CLJ_TOOLS_DIR - all in the Dockerfile RUN commands - and then have future bb calls use that when the container is running.

why not set DEPS_CLJ_TOOLS_DIR to some predicable location, always?

so first set the env var, then download with bb prepare

oh, you're right

! 😄

I think that the problem lies in various containers running on the Jenkins agent (or whatever it's called) and they are not all guaranteed to know the same files.

Learning a lot about our Jenkins setup these days, but it's very slow to iterate changes onto. It's a big mistake that we can't run it locally. Soooo slow.

You will need to fetch the files you need when you switch agent e.g. do a source checkout again. In kubernetes you would have a new pod with a new agent if that is what you use.

I catch what you're saying. The switch is not explicit in our Jenkins.k8s script, which is why I tried to bake in bb prepare to the Dockerfile - I thought I could preempt the problem. I still think I can, I just need to follow @gaverhae’s note about RUN and runtime users.

Or, at least I need to try that out 🙂

Part of the problem is also that I don't have the keys to the murkiest machine room - another team owns the infrastructure. But I was fortunate to get help from one of them today and got some great insights.

yeah, a repro would be helpful

to work around the issue, perhaps including a newer version of fipp would help

I guess the question is include it where? if i specify latest in my extra-deps it doesn’t help 🤔

if you execute your task with --debug you should see some babashka.deps/add-deps call at the top, this will maybe help

bb --debug your:task

so some more info - the lib I’m pulling in does this in core.clj:

(mi/instrument!)
((requiring-resolve 'malli.dev/start!))

perhaps you're pulling in an older version of malli which depends on fipp?

I’m on latest malli - and I’ve specified latest fipp in my deps It’s the fact that it only happens if I split my deps between top level deps and extra-deps in the task that confuses me the most

sure. is it possible to still share the babashka.deps/add-deps call it produces when it goes wrong?

sure:

(babashka.deps/add-deps '{:deps {com.kroo/infreify {:local/root "../infreify"}, fipp/fipp {:mvn/version "0.6.27"}}}) 

this infreify lib is what pulls in malli and calls ! on load

you could also print the classpath your tasks uses with:

{:tasks (do (prn (babashka.classpath/get-classpath)) 
            (your-normal-task))}

to see if it pulls in an older version of fipp or malli

OK yes I’m getting 0.6.17, even if malli is 0.19.1 - is there a way with bb to see the deps resolution like you can with deps or lein?

can you try bb -Sforce your-task perhaps?

perhaps it's a caching issue

and which bb version are we talking here?

dep resolution, sadly no

> and which bb version are we talking here? 1.12.204

OK some more useful info which explains the difference. If I have deps + extra-deps, I have 2 versions of fipp on my classpath (0.6.17 and 0.6.27, in that order) - so my guess is that bb resolves each set of deps independently and then just concats them together? So I should be able to work around that by making sure my top level deps also include more recent version of fipp as well

a repro would be useful here. For this bb.edn

{:deps {fipp/fipp {:mvn/version "0.6.17"}}
 :tasks {a {:extra-deps {fipp/fipp {:mvn/version "0.6.27"}}
            :task (prn (babashka.classpath/get-classpath))}}}

$ bb --debug a

(babashka.deps/add-deps '{:deps {fipp/fipp {:mvn/version "0.6.27"}}}) ;; deps

"/Users/borkdude/.m2/repository/fipp/fipp/0.6.17/fipp-0.6.17.jar:/Users/borkdude/.m2/repository/org/clojure/core.rrb-vector/0.0.14/core.rrb-vector-0.0.14.jar:/Users/borkdude/.m2/repository/fipp/fipp/0.6.27/fipp-0.6.27.jar:/Users/borkdude/.m2/repository/org/clojure/core.rrb-vector/0.1.2/core.rrb-vector-0.1.2.jar"

So the top level dep stays on front of the classpath and is loaded first. add-deps simply concatenates more stuff

Yeah that makes sense - but in this case doesn’t it mean it’s likely to use the first one that’s loaded?

In this case fipp 0.6.17 will be used

yeah, if you already load the dep, it won't be reloaded

yeah it makes sense now - do you know if this is how extra-deps behaves when using aliases in deps.edn?

I don't think that's how extra-deps behaves in deps.edn. I could probably try to make bb behave closer to that

yeah I mean it would avoid this specific type of edge case, don’t know how complicated it would be to modify and whether it’s worth the effort 😄

I'm still interested in the repro though, just to understand the issue. since you're not specifying an older malli anywhere how does it get on the classpath for example.

and didn't -Sforce help

ah sorry your example is a good repro of the main issue which is another version of fipp on the classpath - in my case it turns out the lib in the top level deps pulls in babashka itself - so the same code can be run in clojure I think, and actually just for fs and processso I’ll probably pull those in explicitly instead babashka pulls in nubank/matcher-combinators 3.6.0 (in test profile?) which pull in midje/midje 1.10.6 which pulls in mvxcvi/puget 1.1.2 which pulls in fipp/fipp 0.6.17

here’s a repro:

{:deps {babashka/babashka {:mvn/version "1.12.204"}}
 :tasks
 {my-task {:extra-deps {metosin/malli {:mvn/version "0.19.1"}}
           :requires ([malli.core])
           :task (do ((requiring-resolve 'malli.dev/start!))
                     (println "running my task!"))}}}

babashka doesn't depend on matcher combinators, at least not directly. it's not recommended to directly depend on babashka for bb scripts. it's not intended to be used that way. you can use bb print-deps to get deps.edn with dependencies that you can run in the JVM

so please stop doing that :)

I mean it’s imported in the underlying lib that’s pulled in by the script, not the script itself, in my case

babashka's deps.edn top level deps contain a lot of crap that is only used for testing

but yeah, I’ll remove the dep on babashka in the other lib

bb print-deps is cool though, didn’t know about that 👍