How long does builds take for you folks in average? I am being told node builds take 5-10 minutes and I want to make sure I am not doing anything wrong before answering. At the moment a monorepo build takes 22 minutes and it downloads ~50 deps

This is more of a #clojurescript question actually

First of all, I know tools.deps/clj is not meant as a build tool. But still: We now have a bunch of utils like pack, Eastwood, depot, testrunner, etc which are useful tools to use in a build pipeline. I know I can add aliases to all of these in my home-directory’s deps.edn, but that doesn’t really work across multiple machines. Having all the aliases duplicated across multiple projects doesn’t seem like a great solution either. One solution I could think of is that deps.edn could refer to other deps.edns and use their aliases. Would that be possible?

what exactly is the problem in duplicating between projects? lack of standardization?

I have stuff like this:

:ancient {:extra-deps {olical/depot {:mvn/version "1.2.0"}}
                     :main-opts ["-m" "depot.outdated.main"]}

you want to make that implicit in your deps.edn?

I was imagining that when I wanted to bump the version on olical/depot, I’d have to check out all my projects and bumpt it.

Also, some of the aliases are somewhat more involved than this.

yup. it’s a difficult trade-off. it’s tempting to standardize or “dedupe” some of this boilerplate, but that creates problems when things aren’t as equal anymore

e.g. you loose the flexibility

I’ve come to realise this. But that’s another discussion we could have over a cup of coffee IRL 🙂

if you have a lot of very similar projects I guess you could make your own dev package, and encode the aliases in it, and then use it like so:

{:aliases
 :dev {:extra-deps {my.company/my-bundle {:mvn/version "1234"}}}
 :test {:main-opts ["my.bundle.test"]}
 :ancient {:main-opts ["my.bundle.ancient"]}
 :build {:main-opts ["my.bundle.build"]}}

yes, for sure 🙂

but that approach does quickly leads to entangled framework world, use with caution

I guess so.

I don’t have enough similar projects to warrant the effort, but this is an approach I would consider to contain the duplication and lightly enforce some standardization. you’d still have to bump the shared dependency in all projects, but at least you reduced n deps/aliases to 1

I’m sure there are other ways to approach this as well

One thing I noticed really is that when most of those things are just standalone utilities, they shouldn’t even be in my main deps.edn file, because they don’t need all my “production” stuff to run.

I’m always thinking that perhaps it’s time for a new “dev-time” tool that provides a wrapper and some porcelain around those single-purpose libraries. Sure you lose the flexibility, but you also don’t have to make all those decisions when you are a newcomer to the ecosystem, plus the single-purpose libraries can still be used standalone and can evolve at their own pace.

wherever it is, all the deps and scripts required to build and deploy the app should reside in the repo at least

@orestis would such a thing be put into each project? Or be part of the setup guide for ~/.clojure/deps.edn when you're new to a project?

I strongly believe that developing any app should require as few pre-installed tools as possible, and that the code/repo should be as self-contained as possible

I think it could be its own standalone CLI, like lein or boot.

But in the end all it would do is generate the correct clj/clojure invocation for you.

I’m currently using deps.edn + Makefile to get the job done. I see some things that will probably seem like tedious repetition at some point, but so far I’m loving the simplicity of it. no magic, you can see everything

I see your point though @christian767 — though with tools.deps you need to have clj installed — otherwise, well, you have to fallback to java -cp ..., no?

yes, I think clj and java is a good baseline requirement for clojure projects at this point

Right, a more opinionated Makefile is what I’m trying to describe.

sure

My experience in trying to setup a new deps.edn based project was a few days trying out the various uberjar libraries, then a linter, then a formatter, then an outdated-deps-finder etc etc etc.

I think the community would benefit if this is done for you, while still maintaining the much-needed transparency and zero magic.

I agree

I wonder how could this move forward? Ask people to upload their deps.edn and makefiles and shell scripts somewhere and compare approaches? I’m too much of a newb to do this based only on my assumptions, and different projects will have wildly different needs.

@orestis JUXT are solving this with https://github.com/juxt/edge

You start a project by cloning edge, and then modifying it. Everything is set up for you. clj is all you need.

Templates also solve this problem.

basically, scaffolding

I wouldn't say scaffolding is a good comparison. It's more of a cookie cutter or starting point than anything.

The issue with templates/scaffolding is that after the initial project creation, you are on your own.

my issue with this approach is that it sets you off for an overwhelming start, and it is too easy to end up with a bunch of stuff laying around that you don’t really need

@orestis that's why we suggest cloning. You can upgrade by doing git merge edge/master

I've been making some adjustments to reduce the chances of conflicts to very low. It should only happen if you modify things (in which case, what do you expect?)

The documentation around this isn't great, but it's how we expect edge to be used, and how we use it internally.

blugh, s/modify things/modify shared areas in a separate folder/

I see your point — and it might be hugely valuable for a consulting company, since you tend to start projects very often, right? I’m more after something that decouples some very common concerns for most people, while imposing no opinions on how your actual app is structured. Meaning, I just want to run a command to give me outdated dependencies, a command to run all my tests, a command to build my uberjar. I’d like newcomers to the community to have a single entry point, and if/when they want more power, be able to drop down to the layer below.

Crucially, they shouldn’t have to know the underlying libraries if they don’t care. Similarly to how you don’t really know how lein builds uberjars.

I know 🙈 You don't want to

And also crucially though, the uberjar building library is standalone and people can use it as they see fit.

koacha does this for tests

you could create a binstub on your box, run koacha in your app, and it’ll “just work”

We start, and maintain, a lot of projects, yep 🙂. edge is slightly more opinionated than that, it assumes integrant for example. But I think the implementation would work for this? Use a git repo, and merge periodically to have the uberjar builder updated by maintainers upstream.

Exactly @christian767, kaocha is very good at that, it has its own CLI - like thing, but you can also use its API if you want to.

yup

Binstubs are only part of the puzzle though 🙂 What would provide: > I think the community would benefit if this is done for you, while still maintaining the much-needed transparency and zero magic. ?

So what I’m after is — a cli tool that you could install, then run XXX test and it would do the right thing, using kaocha under the hood. Kaocha then can keep evolving and people can use it standalone without the XXX tool.

You wouldn’t know that kaocha is used unless you really want to — kaocha is a very good example since it plays well with clojure.test. Uberjars can be more fiddly to configure, I guess?

This is sort of the discussion I was looking for

But in one way, I guess we’re discussing yet another lein or boot

jars need a lot of configuration in my experience 🙂 Clients come up with all sorts of novel environments in which we have to deploy.

@slipset I wouldn't like to see another lein or boot fwiw.

I know, but the ideas I’m sort of looking for very quickly lead in that direction, unfortunatly

Even if it’s only @christian767 Makefile or Juxt’s edge.

I could imagine ending up in:

$ make -f /usr/local/etc/default-makefile-for-app setup

which is basically lein new app or clj -A:clj-new app ...

or some sort.

I’m not 100% sure if this only an “ergonomics” thing — or if some plumbing is still missing from tools.deps.

Given Stu's advocacy for CLJ-2373 (I've memorized it he's said it so much!), let's take a leaf out of the Clojure/core approach to solving problems. What is the problem statement here?

If we were only trying to solve the assembly problem, then templates are fine. So there's another element too I'm guessing?

I kinda get the impression we're each looking for different additional properties. I've seen mention of "project-local isolation", "baseline as clj", and "not caring about underlying library".

fwiw, I care more about "baseline as clj" because it's the minimum I can expect anyone to have done, ever. If we're doing training as a company, getting binstubs installed, covering all the distros is more hassle than it's worth for me. I would probably find it acceptable if a project had some kind of alias like clj -A:tool-name which would then do the rest for itself. add-lib is pretty compelling here, as the tool can dynamically fetch it's dependencies as needed.

Nice breakdown. Fully agree.

Possibly relevant: https://www.reddit.com/r/Clojure/comments/9tj6d1/katamari_a_clojure_build_tool_a_talk_by_reid/

I uploaded my https://github.com/seancorfield/dot-clojure as a guide for others some time ago -- I think sharing these is useful while the tool is still new.

As for a self-contained project: yes, very important -- it's why we have a shell wrapper that sets CLJ_CONFIG (to point to a repo-specific "dot Clojure" folder) and we have a deps.edn file for each subpar object in the repo.

Funny I was doing the same yesterday but it seemed like a hack ^

But today, it no longer seems like you are hacking. Today, you are corfielding 🙂

It seems like everybody is going to corfield their deps.edn sooner or later 😄

Sean, if this goes seriously weird, I sincerely apologize

Back to the CLJ_CONFIG stuff (I was on my phone earlier)... we've found the ability to specify a "home" deps.edn via CLJ_CONFIG is great! It means a project can have a "home" configuration for aliases that provide project-specific tooling as well as :override-deps (which we rely on heavily to "pin" versions across subprojects within the overall project.

I talked a bit our shell wrapper a week or two back I think. Mostly it takes commands of the form build alias(es) subproject and it goes into that subproject and runs CLJ_CONFIG=../versions clojure -A:defaults:aliases (edited to add our defaults alias -- very important!)

Is that only for aliases?

It does a few mini-DSL-y sort of things around plural aliases or aliases with certain prefix/suffix strings as well as expanding some internal "aliases" but it's only a few tens of lines of shell script.

@alexmiller We have the "pinned" version of Clojure specified as a regular :deps in that versions/deps.edn file but everything else is just aliases I think.

Oh, and it adds our Archiva Maven repo as well.

It's definitely shown the need for all deps-based tooling to support correct deps.edn merging with support for :override-deps and aliases, if it doesn't just execute in the context of your project's classpath (Depot, for example, already supports aliases but needed a PR to support :override-deps merging).

It would definitely simplify usage if :override-deps could be top-level and not need an alias.

@seancorfield fwiw, I don't like the CLJ_CONFIG solution because it takes away the ability to have aliases (e.g. A custom repl) from a user. I understand your choice and think it is pragmatic though.

Yeah, it's a compromise. We have REPL-related aliases in that deps file, as well as other tooling.

What I'd really like would be the ability to specify an extra deps.edn file easily between user-deps and project-deps.

So we'd have <system>/deps.edn, <user>/deps.edn, <tooling>/deps.edn, (project) deps.edn.

Say, I'm getting this error trying to mvn install a tools.deps.alpha repo I've got checked out locally, anyone know if there is some special stuff I need to do in order to build it?

[INFO] --- clojure-maven-plugin:1.7.1:test (clojure-test) @ tools.deps.alpha ---
Error: Could not find or load main class clojure.main
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

What are people using as an environ replacement? Does it only work with a project.clj?

We have our own, internal, configuration library that can use env vars but mostly relies on EDN files.

We've considered open sourcing it, but it does contain some proprietary stuff that we haven't had time (or inclination) to refactor out yet.

If I have this in my :deps:

compute/integrations {:git/url    "[email protected]:ComputeSoftware/integrations.git"
                      :sha        "e0b915e5e9b285d322b6d01231d8e4e468550d76"
                      :exclusions [commons-logging/commons-logging]}

compute/integrations [email protected]:ComputeSoftware/integrations.git e0b915e
  com.amazonaws/aws-java-sdk-autoscaling 1.11.399
  com.amazonaws/aws-java-sdk-sts 1.11.399
  com.cognitect/anomalies 0.1.12
  clj-http/clj-http 3.9.1
    org.apache.httpcomponents/httpasyncclient 4.1.3
      org.apache.httpcomponents/httpcore-nio 4.4.6
    slingshot/slingshot 0.12.2
    org.apache.httpcomponents/httpcore 4.4.9
    org.apache.httpcomponents/httpclient-cache 4.5.5
    org.apache.httpcomponents/httpclient 4.5.5
      commons-logging/commons-logging 1.2

dunno, that’s weird

feel free to file a ticket with a repro if you can slim it down

it being under a git dep might be part of the answer, not sure

Will see if I can get a repro.

First cut at the obvious repro case and I cannot reproduce. I made a git repo with a deps.edn that has clj-http as the dep. I made directory with a deps.edn on my computer that added that lib as a dep with the commons-logging exclusion and it behaved correctly. Is there something else I should be looking for in my current problem case that may shed some light here?

If I add that exact dependency to my new project:

{:deps {compute/integrations {:git/url    "[email protected]:ComputeSoftware/integrations.git"
                              :sha        "e0b915e5e9b285d322b6d01231d8e4e468550d76"
                              :exclusions [commons-logging/commons-logging]}}}

Ok this is bizarre. After running clj -Stree in my original project after making no changes, I no longer have this issue. I ran clj -Stree -Sforce before so I would've thought that would cover any caching issues. Never mind, I searched the wrong terminal output.

This dep excludes httpclient, as expected.

compute/integrations {:git/url    "[email protected]:ComputeSoftware/integrations.git"
                      :sha        "e0b915e5e9b285d322b6d01231d8e4e468550d76"
                      :exclusions [org.apache.httpcomponents/httpclient]}

Is there a way to see all the deps that have commons-logging as a dep?

Removing all other deps in my project besides Clojure and compute/integrations causes the exclusion to work correctly.

Got a repro. There is something seriously funky going on. Repro is here: https://github.com/kennyjwilli/clj-exclusions. cd into the app directory and run clj -Stree -Sforce. I get this:

org.clojure/clojure 1.9.0
  org.clojure/core.specs.alpha 0.1.24
  org.clojure/spec.alpha 0.1.143
integrations/integrations ..
  clj-http/clj-http 3.9.1
    commons-codec/commons-codec 1.11
    org.apache.httpcomponents/httpasyncclient 4.1.3
      org.apache.httpcomponents/httpcore-nio 4.4.6
    slingshot/slingshot 0.12.2
    commons-io/commons-io 2.6
    org.apache.httpcomponents/httpcore 4.4.9
    org.apache.httpcomponents/httpclient-cache 4.5.5
    org.apache.httpcomponents/httpclient 4.5.5
      commons-logging/commons-logging 1.2
    potemkin/potemkin 0.4.5
      clj-tuple/clj-tuple 0.2.2
      riddley/riddley 0.1.12
    org.apache.httpcomponents/httpmime 4.5.5
com.amazonaws/aws-java-sdk-cloudwatch 1.11.399
  com.amazonaws/jmespath-java 1.11.399
    com.fasterxml.jackson.core/jackson-databind 2.6.7.1
      com.fasterxml.jackson.core/jackson-core 2.6.7
      com.fasterxml.jackson.core/jackson-annotations 2.6.0
  com.amazonaws/aws-java-sdk-core 1.11.399
    joda-time/joda-time 2.8.1
    com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.6.7
    software.amazon.ion/ion-java 1.0.2

org.clojure/clojure 1.9.0
  org.clojure/core.specs.alpha 0.1.24
  org.clojure/spec.alpha 0.1.143
com.amazonaws/aws-java-sdk-cloudwatch 1.11.399
  com.amazonaws/jmespath-java 1.11.399
    com.fasterxml.jackson.core/jackson-databind 2.6.7.1
      com.fasterxml.jackson.core/jackson-core 2.6.7
      com.fasterxml.jackson.core/jackson-annotations 2.6.0
  com.amazonaws/aws-java-sdk-core 1.11.399
    joda-time/joda-time 2.8.1
    com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.6.7
    software.amazon.ion/ion-java 1.0.2
    org.apache.httpcomponents/httpclient 4.5.5
      commons-logging/commons-logging 1.2
lib/lib ..
  clj-http/clj-http 3.9.1
    commons-codec/commons-codec 1.11
    org.apache.httpcomponents/httpasyncclient 4.1.3
      org.apache.httpcomponents/httpcore-nio 4.4.6
    slingshot/slingshot 0.12.2
    commons-io/commons-io 2.6
    org.apache.httpcomponents/httpcore 4.4.9
    org.apache.httpcomponents/httpclient-cache 4.5.5
    potemkin/potemkin 0.4.5
      clj-tuple/clj-tuple 0.2.2
      riddley/riddley 0.1.12
    org.apache.httpcomponents/httpmime 4.5.5

the ordering is somewhat dependent on traversal order as to which one is encountered first, but you shouldn’t be getting a dep that is excluded

there are a couple tickets out there that I’ve looked at and there are a couple issues to resolve - it’s likely this is the same as one of them

Ok. Which tickets?

They're only different because org.apache.httpcomponents/httpclient is coming in via two dependencies, not just one.

In the integrations case, httpclient and logging come in via clj-http. In the lib case, httpclient and logging are coming in via aws cloudwatch (via aws core)

The inconsistency is quite misleading though.

TDEPS-58 (and TDEPS-79)

If you have the exclusion on clj-http, it won't stop it being pulled in if the cloudwatch dep is walked first -- since it won't have seen the exclusion at that point.

We've had to be very aggressive with exclusions to stop up all the "holes" where certain things leak in...

there have been some requests for “global” exclusions

Aggressive meaning exclusions on a lot of your deps?

I’m not necessarily opposed to that, but haven’t thought through all the implications

I think Boot has that.

Yes. We've had to add the same/similar exclusions on several deps across our subprojects in order to completely "hide" them.

Would global exclusions solve your problem?

We've used global exclusions with Boot. It's a bit of a sledgehammer. We used it to exclude Clojure and ClojureScript across everything.

Yes, global exclusions would "solve" the problem but it's really a huge hammer to crack a little nut. And the fall out is that you then need to ensure you bring in all those things explicitly in the right subprojects -- which is a pain.

seems like that would make it hard to use Clojure or ClojureScript :)

You can bring them in at the top-level. It just ensures nothing else brings them in.

Dependency resolution wouldn't solve that for you?

with Maven dependency, no

Maven takes the first version it encounters

Two things: 1. we wanted ClojureScript excluded completely for all our backend projects, 2. we run "pedantic" style checks (or, at least, we did in lein and boot) so if potentially conflicting versions came in, we would get an exception -- and we didn't want to add those exclusions to nearly every Clojure dependency, since everything depends on different versions of Clojure out there 🙂

Anyway, thanks for the help. Excluding commons-logging from my other top level AWS deps fixed the issue. I voted and followed TDEPS-58.