Fork me on GitHub
#tools-deps
<
2022-12-22
>
Adam Helins10:12:49

CI started failing like this after some changes in dependencies and little things. Could it be that we screwed up $HOME or something like that? :thinking_face:

Error building classpath. Can't create directory: /.clojure/.cpcache

practicalli-johnny11:12:08

Suggest checking the value of :config-dir and other settings and see if it got set to /

❯ clojure -Sdescribe
{:version "1.11.1.1208"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/practicalli/.config/clojure/deps.edn" ]
 :config-user "/home/practicalli/.config/clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/practicalli/.config/clojure"
 :cache-dir "/home/practicalli/.cache/clojure"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}
If so it could be via HOME , XDG_CONFIG_HOME or CLJ_CONFIG environment variables (although it would be pretty obvious in the terminal if HOME was not correct)

👀 1
Adam Helins12:12:02

Yes, I kinda forgot about -Sdescribe. I spotted a discrepancy between clojure -Sdescribe and bb clojure -Sdescribe which was really helpful. Thanks!

👍 1
rickmoynihan12:12:26

I’m trying to create a multi-module build with tools.deps/`tools.build`. I have some sub modules (deps) which are also individually maven artifacts, and I want to also build a parent artifact that simply expresses dependencies on all of the sub module deps. Does anyone know of any simple examples, or ways to do this?

rickmoynihan12:12:35

My parent deps.edn currently looks something like this (so I can get a single repl for all sub projects together):

:deps {project/submodule-a {:local/root "submodule-a"}
           project/submodule-b {:local/root "submodule-b"}
        }
Obviously this isn’t quite what I want, as when I create a basis from this it inlines those projects into the parent dep. I’m assuming (and currently looking into) creating the basis (and ultimately jar) dynamically in my build.clj like this:
(b/create-basis {:project {:deps {'project/submodule-a {:mvn/version version}
                                                  'project/submodule-b {:mvn/version version}}}}})
But then I think I’ll need to have installed those submodules into .m2… is there a way to avoid having to pollute .m2 in order to just build the candidate parent dep? I think it would be nicer to separate building from installation if possible.

rickmoynihan13:12:17

hmm also just realised… some submodules depend on each other; and I’d wired up those submodules to use :local/root which works fine at a repl; but it does mean that they are also individually being inlined locally. So I suppose I need to override them too when I create those basis 😞

Alex Miller (Clojure team)13:12:31

What is the end goal of the build? Do you want individual artifacts for each subdirectory with Maven dependencies between them?

rickmoynihan13:12:32

@alexmiller: End goal is to build a set of libraries where: 1. there are individual library artifacts for each subdirectory with Maven dependencies between them 2. a parent library artifact with maven dependencies on all sub projects

Alex Miller (Clojure team)13:12:25

And presumably 3. Work with them locally without artifacts

rickmoynihan13:12:35

yes… well it depends what you mean by that

Alex Miller (Clojure team)13:12:49

3 is easy with local/root and 1/2 are easy with maven/version but it’s hard to do both. You can use aliases with override-deps to swap out but alias application is shallow

rickmoynihan14:12:42

Ok I think that might explain why this:

(b/create-basis {:aliases [:overrides]
                :extra {:aliases {:overrides
                                   {:override-deps
                                   {'myns/module-a {:mvn/version version}
                                    'myns/module-b {:mvn/version version} 'myns/module-c {:mvn/version version}}}}}})
isn’t doing what I’d expect (when fed into jar) it’s including src paths it shouldn’t.

rickmoynihan14:12:19

I wonder if I’d have more luck reading the deps.edn as data and replacing the :local/root s myself?

Alex Miller (Clojure team)14:12:24

you can use :local/roots in the whole thing, consume with :local/roots and just do the replacements in a custom build script for deployment

rickmoynihan14:12:04

yeah that’s basically what I’m trying to do, from a build.clj at the project root; but the internal deps are :local/root’s. First I tried it with with :extra :overrides, but it doesn’t seem to work — presumably because the application is shallow?! So now I’m looking at basically doing something like this, from the top:

(b/create-basis {:project (-> (edn/read-string (slurp "submodule-a/deps.edn"))
                                              (update-in [:deps 'myns/submodule-a] (constantly {:mvn/version version})))})
,,,

Alex Miller (Clojure team)14:12:55

I'm not sure what :extra :overrides means there - :extra is an extra deps.edn (like -Sdeps on the command line)

Alex Miller (Clojure team)14:12:55

:override-deps should apply throughout the tree though

rickmoynihan14:12:35

yes I realised that — I did it because :project is set to the submodules “deps.edn” (which doesn’t have the :override-deps alias in it)… And it looked like :override-deps only has meaning in an :alias (not at the top of a deps.edn map) Though not 100% sure it’s right

Alex Miller (Clojure team)14:12:44

as deps.edn becomes more complicated, things like slurp / read-string are going to be less likely to work and it will be better to lean on tools.deps apis (like https://clojure.github.io/tools.deps/#clojure.tools.deps/slurp-deps)

rickmoynihan14:12:29

ah ok, thanks for the heads up there 🙇

Alex Miller (Clojure team)14:12:49

> :override-deps only has meaning in an :alias correct

Alex Miller (Clojure team)14:12:24

if you are "changing contexts" in a deps.edn, you'll want to use the tools.build apis for that

rickmoynihan14:12:16

yes I’m already using set-project-root

Alex Miller (Clojure team)14:12:50

I guess, I don't understand why the override thing "doesn't work"

rickmoynihan14:12:03

though I wrote a macro to help with it:

(defmacro with-project-root [dir & forms]
  `(let [orig-root# @#'b/*project-root*] ;; yes this is correct, my cat didn't run over my keyboard I promise! :-)
     (try
       (b/set-project-root! ~dir)
       (let [ret# (do ~@forms)]
         ret#)
       (finally
         (b/set-project-root! orig-root#)))))

Alex Miller (Clojure team)14:12:31

in particular, I don't understand exactly what you tried and what you saw and how it differed from what you expected

Alex Miller (Clojure team)14:12:29

seems like rather than with-project-root, you could more directly just bind *project-root*

Alex Miller (Clojure team)14:12:46

(defmacro with-project-root [dir & forms]
  `(binding [b/*project-root* ~dir] ~@forms))
should be basically the same right?

rickmoynihan14:12:25

Yes, I did actually have a question about that — I guess I was surprised to you setting the var directly, rather than suggesting users bind it (which was my first instinct); so I just assumed you had good reasons to use set. But thanks for clearing that up for me! 🙇

Alex Miller (Clojure team)15:12:11

I was expecting people to change context as a 1-time kind of thing, not as a per-submodule change-and-reset, which in retrospect is an obvious thing. guess I should just add with-project-root to t.b

👍 1
rickmoynihan15:12:48

> in particular, I don’t understand exactly what you tried and what you saw and how it differed from what you expected So far everything I’ve tried has had the same effect, which is that the src paths for the independent module jars are being unioned… e.g. if I have in submodule-c:

{:deps {myns/submodule-c {:local/root "../submodule-a"}, myns/submodule-b {:local/root "../submodule-b"}}}
Then my output jar for submodule-c contains :paths from a and b (along with their pom.xml’s etc). Of course I’m currently just assuming this is because of :local/root (I guess I could be crossing the streams somewhere else)

rickmoynihan15:12:41

I haven’t yet tried just replacing those myself, by slurping and reading the file and munging the data, but that’s what I’m about to try next.

Alex Miller (Clojure team)15:12:44

being unioned ... where?

rickmoynihan15:12:12

I mean the jar contains the union of :paths from the submodules referenced via :local/root

rickmoynihan15:12:58

(Not an explicit call to set/union or anything like that)

Alex Miller (Clojure team)15:12:29

instead of just trying stuff, let's just be really clear about what you're doing

Alex Miller (Clojure team)15:12:47

so in submodule-c, you have a deps.edn like:

{:deps {myns/submodule-a {:local/root "../submodule-a"}
        myns/submodule-b {:local/root "../submodule-b"}}

Alex Miller (Clojure team)15:12:12

you're trying to create a jar for c that references a and b as maven artifacts, not as locals

rickmoynihan15:12:22

yes, but submodule-c is not the parent (the parent also may shortly contain a submodule-d)

Alex Miller (Clojure team)15:12:16

you control everything that is being used to make the jar

Alex Miller (Clojure team)15:12:49

you can copy the source from c to your jar making location, so that's fine

Alex Miller (Clojure team)15:12:13

you control the basis used with write-pom which defines which deps the jar is published with in the pom

Alex Miller (Clojure team)15:12:15

so if you provide an :extra deps that replaces the top-level deps, that's all you need to do. you don't care about the transitive deps - they don't affect what's in the jar

Alex Miller (Clojure team)15:12:14

so when you create-basis, use :extra {yourgroup/artifact-a {:mvn/version v}, yourgroup/artifact-b {:mvn/version v}}

rickmoynihan15:12:29

> you can copy the source from c to your jar making location, so that’s fine I could — but I’m not currently copying src files around myself.

Alex Miller (Clojure team)15:12:31

I mean like (b/copy-dir {:src-dirs ["src" "resources"] :target-dir class-dir})

rickmoynihan15:12:58

> so when you create-basis, use :extra {yourgroup/artifact-a {:mvn/version v}, yourgroup/artifact-b {:mvn/version v}} This was the first thing I tried…. I think it was actually (b/create-basis {:extra {:deps ,,,}}) before trying the form above with override-deps. Also another reason I moved to try again with override-deps was that I’m also building module-a and module-b, and I didn’t want to maintain separate overrides for each… i.e. I thought the semantics of override-deps were “if a dep uses this coordinate in its deps.edn override it; but if it doesn’t mention it don’t introduce it”

rickmoynihan15:12:39

> why > I mean like (b/copy-dir {:src-dirs [“src” “resources”] :target-dir class-dir}) I could do that too; but I was avoiding that by doing it in situ by changing b/*project-root*

Alex Miller (Clojure team)15:12:53

you still have to do something to get the sources

rickmoynihan15:12:52

Does org.corfield.build/jar with the basis not handle that?

Alex Miller (Clojure team)15:12:55

you have code - why maintain anything? just transform the data you have in the deps.edn

Alex Miller (Clojure team)15:12:10

this is the first you've mentioned using that

rickmoynihan15:12:18

> you have code - why maintain anything? just transform the data you have in the deps.edn yes that’s the approach I’m going to try next

rickmoynihan15:12:35

> this is the first you’ve mentioned using that Sorry.

Alex Miller (Clojure team)15:12:38

tbh, I don't like Sean's wrapper b/c it only works until something is different, then you have to fall back to using tools.build. the whole idea of tools.build is to start with something that can grow

Alex Miller (Clojure team)15:12:11

this is a case where a lot of things are different

rickmoynihan15:12:32

yeah I see it does the copy-dir

Alex Miller (Clojure team)15:12:59

you're going to twist yourself in knots to create a context where his thing does the one thing it does, or you can just do it directly with tools.build. I think in the end you'll understand more and be able to do the next thing better with the latter

rickmoynihan15:12:27

Yeah, that’s fair… I’ve used tools.build before, but also found seans wrapper useful on simpler builds. However I do also see that the copy-dir inside seans jar is causing my problem, because it doesn’t clean the :target-dir between runs.

rickmoynihan15:12:50

So I think you’re right, that this is the root cause

rickmoynihan15:12:27

Thanks for the help 🙇

rickmoynihan15:12:26

Another case of wanting simple not easy 🙂

seancorfield17:12:39

@U06HHF230 A suggestion for you: look at the structure that Polylith uses. It has a development "project" for REPL usage where all code is on the classpath via :local/root (using a :dev alias) and then it has projects folders -- one for each buildable artifact -- with deps.edn that say how to build that specific artifact. It's primarily intended to build applications (so inlining :local/root deps is acceptable there) but I think it could also be used to build independent libraries, if you can figure out a model for how you version them all (and therefore build them in reverse dependency order, bottom-up, which should be reasonably straightforward?).

rickmoynihan17:12:51

Thanks… I’ve been purposefully avoiding polylith at the minute; simply because it seems larger than I need, and more specialised towards application mono-repos.

seancorfield17:12:25

> However I do also see that the copy-dir inside seans jar is causing my problem, because it doesn’t clean the :target-dir between runs. That is deliberate, so you can combine jar with other actions. As Alex says, that wrapper is fine for the simple stuff but breaks down in more complex settings -- at work we use a mixture of direct tools.deps, tools.build, and a few build-clj wrapper functions... and I'm planning to wean us off it "soon" as well as updating at least deps-new to use raw tools.build instead of the wrapper.

seancorfield17:12:05

(well, Alex doesn't say the wrapper is "fine" but I hope you get my meaning)

rickmoynihan17:12:21

yeah I realise it’s fine for bb/jar’s intended purpose… just not mine (here) 🙂

seancorfield17:12:56

And, yes, the wrapper was born out of an apparent desire for "easy" over "simple" 🙂

rickmoynihan17:12:42

as I said I’ve used the bb wrapper before, and found it helpful at cutting down boilerplate. I was also assuming when I set out that there might be an easy solution to this; which was partly why I started with your wrapper.

rickmoynihan17:12:26

I think I’ve said this to you before though… I think a better way to slice “easy” is vertically; separating the needs of applications from the needs of libraries. Where as bb assumes it can add ease for both applications and libraries together.

seancorfield17:12:11

Indeed. And it started out as a very "simple" wrapper to just cut down some basic boilerplate... but grew far too many knobs and dials in the pursuit of reducing boilerplate in more cases instead of "encouraging" people to just use tools.build... I'm far too accommodating in that area 🙂

rickmoynihan17:12:22

haha well you do yourself a disservice there, there’s certainly a need for easier solutions to common problems… even if they do bring some complecting to the table. I think the problem is that easy solutions naturally grow knobs; and become a bit of a mess and hard to maintain overtime… Simple ones, more purposefully leave problems on the table, but are more maintainable over the long term. Anyway — I do very much appreciate everything you’ve done 🙇

rickmoynihan17:12:41

I would have still liked an easy solution for this; as it’s taking far longer than I was expecting… and I was expecting it to take a while too! :rolling_on_the_floor_laughing:

Alex Miller (Clojure team)17:12:42

I added with-project-root for the next t.b release btw

👌 1
seancorfield17:12:04

Heh, well, you've probably seen my long series of blog posts about our monorepo and the various approaches we've taken over the years, and the various things we've had to solve even with Polylith in the mix...

rickmoynihan17:12:52

Yes, I’ve returned to them many times over the years

Alex Miller (Clojure team)13:12:36

So not sure if that’s sufficient (depends on your deps)

rickmoynihan13:12:52

3.1. I’d definitely like to work across the whole library as a repl (this bit currently works via :local/root)

rickmoynihan14:12:04

3.2. (optionally) it would be nice to consume the library and its individual bits without artifacts

rickmoynihan14:12:18

consume from an application outside of the libraries mono-repo setup… i.e. through a :override-deps {:local/root ,,,} in an alias in my ~/.clojure/deps.edn

rickmoynihan14:12:30

> 3 is easy with local/root Are you meaning my 3.1 or 3.2 here?

rickmoynihan14:12:49

I’m guessing 3.1