Fork me on GitHub
#tools-deps
<
2019-02-04
>
borkdude11:02:06

In a mono-repo is it possible to use a common deps.edn file with aliases that can be used from all sub repos?

borkdude11:02:34

(without hacking $CLJ_CONFIG)

dominicm12:02:32

Not that I've discovered.

dominicm12:02:39

I don't believe it's possible.

alexmiller12:02:40

No, not currently

borkdude12:02:42

I’m experimenting with porting https://github.com/borkdude/boot-bundle to deps.edn, so in the future, we could migrate our boot stuff to deps.edn

borkdude12:02:18

The idea is that you can define a bunch (bundle) of deps in a common file and then refer to a common name for that bundle in your deps.edn files’ :deps and/or :extra-deps.

gphilipp13:02:38

@borkdude not sure if that covers your need, but @matthias.margush has created this tool: https://github.com/matthias-margush/aka

seancorfield17:02:25

We have a monorepo with dozens of subprojects -- and we use a small shell script to wrap clojure and use CLJ_CONFIG to point to the monorepo's deps.edn file (and then each subproject has its own deps.edn file). Works great. Any reason you don't want to do that @borkdude?

seancorfield17:02:59

Basically the shell script does cd path/to/subproject.clj; CLJ_CONFIG=../versions/deps.edn clojure ...args...

richiardiandrea18:02:02

datapoint -> we have a monorepo and looking for a clean solution as well

richiardiandrea18:02:28

or hints at how to contribute a solution

borkdude18:02:50

@seancorfield the reason I thought I wanted to avoid that is that it messes with people’s own configs in their .clojure dir?

alexmiller18:02:22

some possible answers, curious if these would help: 1) a clj flag to pass one or more config files that go “in the chain”

alexmiller18:02:44

2) an env property to list or more additional config file locations

alexmiller18:02:35

3) some kind of config file in .clojure/ (I’m guessing probably not ideal as this is for a group of projects but may not extend to everything you’re doing)

dominicm18:02:41

1 makes this manual opt in everywhere, so users have to inform their tooling of this flag. 2 would require something like direnv, and would also require extensive editor integration.

borkdude18:02:26

another way: :include "../parent.deps.edn"

dominicm18:02:29

1 seems more tenable than 2 for something like cursive which has the most extensive analysis of deps.end files.

borkdude18:02:57

That, together with a way of including the deps of aliases into another alias would be helpful to me

ghosttoaster18:02:00

Hey all. I'm having trouble getting a very simple deps.edn going. I keep on getting a message that opencv can't be found in mvn central, but I've added my custom repo where the jar can be found.

eval202018:02:36

1 with less magic: clj --deps-file foo/deps.edn would only take foo/deps.edn into account. The current situation could be seen as clj --deps-file ~/.clojure/deps.edn --deps-file deps.edn ....

seancorfield18:02:05

@borkdude when you're working on a monorepo project, I'd argue you want a more reproducible set of processes so I'd say you want .clojure to be omitted from deps.edn.

dominicm18:02:31

@seancorfield I don't think that's any more true than a normal repo is it?

borkdude18:02:33

@seancorfield people tend to put their cider-nrepl config into .clojure/deps.edn. I don’t want to disturb that

👍 1
borkdude18:02:35

Since deps.edn is “just data”™️ I’m inclined to rewrite a template to the deps.edn that will be used

dominicm18:02:46

I also put Rebel, library overrides for environment specific bugs, benchmarking and more. All depending on the context I'm in.

seancorfield18:02:33

@dominicm It's a matter of opinion 🙂 We've generally added stuff to our versions/deps.edn file that individual devs find useful. And we have a couple of "dev" namespaces that include REPL startup stuff. So there's really nothing in our .clojure/deps.edn files that would be useful in addition (we already have nrepl/socket REPL/REBL stuff in our project deps.edn files).

dominicm18:02:41

@borkdude be aware that updating a deps.edn like that will erase comments. I wrote ednup to be a crappy solution to that, but it's not prime ready yet.

borkdude18:02:09

@dominicm I’m exploring rewrite-clj using zippers, that will leave everything intact. https://github.com/Olical/depot does this too.

seancorfield18:02:25

That said, I wouldn't object to an enhancement to tools.deps/`clj` that allows additional deps.edn files to be specified 🙂

dominicm18:02:26

@borkdude that's what ednup does

borkdude18:02:10

:extra-deps.edns

dominicm18:02:56

@seancorfield I work on a few monorepos (they're mono to their context), and updating my dev config in each of them would be a pain. I'd feel inclined not to commit my dev hacking and clutter git history, especially in projects where we have stringent reviews on code.

dominicm18:02:20

@borkdude yup. You can see usage in my rewrite of resolve tags.

borkdude18:02:47

I’ve started experimenting with supporting deps.edn for boot-bundle here, also using rewrite-clj: https://github.com/borkdude/boot-bundle/blob/tools.deps/src/boot_bundle/rewrite.clj but since it’s hard to settle down on an approach, I’m first looking if deps.edn itself could support this better

dominicm18:02:55

It's not a great library, but I plan to iterate on it intensely at some point, and it's great for simple things.

dominicm18:02:15

Oh, if you want examples, pack has an injector which uses rewrite directly.

borkdude18:02:06

I meant: figuring out the format/API is hard, not the actual programming 🙂

borkdude18:02:41

I want to support existing users of boot-bundle (including myself) and support a migration path from boot to deps.edn

borkdude18:02:21

anyways, being able to specify multiple extra deps.edns from the cmd-line is a good thing in my opinion

borkdude18:02:56

you can always use -Srepro to exclude .clojure so that functionality is already there

alexmiller19:02:46

it is still a pending todo to finalize what -Srepro means, esp in context of -Sdeps and other features. some of that is pending config file rework

hiredman18:02:31

another option, instead of having a super deps.edn for the monorepo, or generating dep.edns for subprojects might be something like a dependency linter, you give it some rules (deps on project x most version y) and run it over a directory and it checks all the deps.edn files it finds

hiredman18:02:08

a linter could even be run against projects not in your monorepo to see how some other library might integrate

borkdude18:02:17

@hiredman meh, then you would still have to update all those occurrences, or do you mean the linter would actually update the files?

hiredman18:02:38

no, you would have to update the files

borkdude18:02:05

the reason I wrote boot-bundle was to prevent having to do that, it became annoying

borkdude18:02:35

I would want one source of the truth and the include that in another project

borkdude18:02:18

some call it “managed dependencies” but I didn’t realize that when I wrote the lib

hiredman18:02:43

how much dependency churn do you have?

borkdude18:02:22

so what I’m searching for is managed deps for deps.edn. Another way to accomplish this maybe is to have “empty” projects that declare their deps and then make a dependency on those with local/root.

borkdude18:02:02

but that gets out of hand too, I want it to be declarative and easy to modify

borkdude18:02:06

@hiredman we have a couple of local libraries which all depend on subsets of the same dependencies. in some of those we use AOT. it’s nasty if you get the versions wrong, so we want them to be the same

alexmiller19:02:18

we always intended to support something like managed deps and I think that is a valuable use case

alexmiller19:02:06

we have not put as much thought into how that should look as with other stuff yet

seancorfield19:02:47

Our versions/deps.edn file is mostly :override-deps to "pin" dependencies for the whole monorepo. Each subproject declares its own dependencies as group/artifact {}. And we have dev/test/build aliases in versions/deps.edn.

borkdude19:02:29

@alexmiller with being able to specify extra deps.edn configs from the cmd line + the ability to merge multiple aliases into one (:foo has :bar and :baz like with -R:bar:baz) I think we would be 99% there

borkdude19:02:36

maybe even 100%

seancorfield19:02:40

Having four deps.edn in the chain instead of three would solve all our issues 🙂 (system, user .clojure, project root, subproject root).

borkdude19:02:10

project/subproject seems arbitrary to me (different projects use different directory structures), being able to specify this yourself and number of times you can do this should be flexible

hiredman19:02:33

aot adds whole other wrinkle on top of things for monorepos, because you are depending on build output of a local project, not the local project (with clj doesn't directly support so much)

hiredman19:02:11

do you aot as low down the deps tree as possible, or as high up as possible?

borkdude19:02:46

@hiredman We wrote a lib that has some Java code which we compile up front. We don’t want to have the situation that we’re going to use different deps than what that was compiled against.

hiredman19:02:27

ah interesting, built java not aot'ed clojure?

seancorfield19:02:30

@borkdude Perhaps I'll rephrase that as "monorepo" and "project" level then instead of "project" and "subproject"? 🙂

borkdude19:02:03

We also ran into trouble once with a service that was running clojure 1.8 which did different edn serialization than 1.9, that was also a problem. we just don’t want to think about this stuff too much and exclude any such possibility.

borkdude19:02:11

@seancorfield boot-bundle supports loading bundles (managed deps) even from Clojars, it doesn’t matter how you organize this.

borkdude19:02:42

@hiredman yes

seancorfield19:02:25

@borkdude and each "bundle" would map to a separate deps.edn file? So you could have a folder for each "bundle" and then access them via multiple :local/root deps?

borkdude19:02:18

@seancorfield how this is done currently in boot bundle is one keyword maps to one or more deps (or recursively, to another bundle). this is just EDN which is expanded dynamically in the scope of a build.boot file.

seancorfield19:02:26

For us, the problem isn't really the deps themselves, but the aliases we want across the whole monorepo -- sort of like requiring every dev to have consistent .clojure/deps.edn files but just for the projects in the monorepo.

borkdude19:02:34

where this bundle.edn file comes from is up to you, could be from clojars, a directory, etc.

seancorfield19:02:22

I haven't looked at anything in Boot for quite a while now. I hadn't heard of boot-bundle. Sounds useful in a deps.edn world.

borkdude19:02:12

in boot this is easy to implement because you can actually program in the build file. in deps.edn I currently see no other way than to rewrite a deps.edn.template and expand it into a deps.edn. but with the options we mentioned in this conversation, I think it’s very close at hand

borkdude19:02:00

relying on add-lib could also work, but I’m not too familiar with that and I think that’s alpha-alpha?

seancorfield19:02:09

I recently added some aliases to my .clojure/deps.edn file that pull in that branch and the comments in my EDN file show how to write a load-master function that can resolve master from git repos and then add that library 🙂

richiardiandrea23:02:17

read the whole conversation, our team is looking for the same answers @borkdude, maybe we can join forces 😄 Share the pain err your thoughts with me ok?