Fork me on GitHub
#tools-deps
<
2020-07-23
>
tengstrand05:07:28

I can set up my deps.edn at the root so that it includes a "module" (that has its own deps-edn) like this:

:aliases  {:dev {:extra-deps {env/dev {:local/root "environments/dev"}}}
           :test {:main-opts ["-m" "polylith.clj.core.cli.poly" "test"]}}
Then I can run: clj -A:test:dev ...and it will run my tests (that is what the 'test' alias does for me). But the deps.edn under environments/dev contains its own aliases with some :extra-paths that I want to include also. Is that possible? I imagine that I need to do something like this (which doesn't work): clj -A:test:dev:dev/local-alias ...where local-alias is defined as an alias in environments/dev/deps.edn:
:aliases {:local-alias {:extra-paths ["../../components/help/src"]}

seancorfield14:07:48

No, aliases in dependencies are not relevant. That would be kind of dangerous as you'd suddenly become vulnerable to whatever aliases developers added to their own projects, which could interfere with your own user-level aliases. And you'd also be affected if a dependency switched from Leiningen to Clojure CLI or vice versa -- which runs completely counter to any notion of reproducibility.

tengstrand06:07:53

Okay, but here I suggest to use existing aliases that lives in environments/dev/deps.edn, that we "import" from deps.edn at the root. Then we don't "mess up things from the outside" just selecting aliases within that "imported" deps.edn. If I "cd environents/dev" first, then I have access to all those aliases which have the purpose to add :extra-paths which I need. But here I have several "sub modules" (not just the one in this example) and wants to do the same from the root deps.edn without having to manually merge all those deps.edn files into one huge deps.edn at the root.

seancorfield06:07:49

As I said @U1G0HH87L "That would be kind of dangerous".

seancorfield06:07:28

:local/root isn't special -- it's just another way to identify a dependency. You're asking for dependencies that include deps.edn to have their aliases combined with the current project. That's not a practical way to deal with reproducibility in project builds.

tengstrand06:07:44

Then I guess I need to solve it with code generation or create my own REPL or similar.

seancorfield06:07:43

What we do at work, with our 100k+ line monorepo containing 30+ subprojects, is that we have a special subproject (called versions) that contains the "master" deps.edn that has all the aliases needed by all projects, including :defaults which brings in all the :override-deps to pin versions etc, and then all the other subprojects really just have :deps and an alias for :test-deps to add anything needed for testing that subproject (`versions/deps.edn` has a :test alias for all the common testing infrastructure). Then we use CLJ_CONFIG=../versions clojure -...aliases in each subproject.

seancorfield06:07:36

CLJ_CONFIG lets you treat a specific folder as if it contains your user-level deps.edn (and so any actual ~/.clojure/deps.edn file is ignored -- which is good: the monorepo then control all dependencies and aliases!)

tengstrand15:07:31

Thanks for the information, can be very useful!

p-himik08:07:01

There's no way to prevent clj from reading deps.edn in the current directory, is there?

borkdude09:07:20

@p-himik don't think so.

p-himik09:07:15

Thanks. That's seems a bit inconvenient because if I want to run some tool with -Sdeps (like Capsule in my particular case) within my project, I cannot really do it without mixing two different classpaths.

plexus10:07:15

it's fairly annoying, I've had scripts that temporarily move deps.edn out of the way because of this...

borkdude10:07:54

deps.clj, an unofficial re-implementation of the clj script does have an option to choose a different deps.edn file than the one in the current directory

vlaaad10:07:23

I think setting :paths in aliases to replace source path from deps.edn is as close as it gets to what you want

borkdude10:07:38

it's also possible to remove deps from the classpath in an alias: https://github.com/borkdude/babashka/issues/152

p-himik10:07:33

But then I'd have to restructure my whole deps.edn just to make some thirdparty tool happy. That's not good.

vlaaad10:07:19

not sure I’m following, can you put that third-party tool as a user-level alias?

p-himik10:07:40

The tool itself - yes. The thing is, it must be run in the presence of deps.edn. But it would be best to not let clj (and I run the tool using clj) also interpret deps.edn.

p-himik10:07:15

The tool itself is used via -Sdeps, so it's not in the deps.edn.

borkdude10:07:44

you can also manually construct the classpath (using a script) and pass it to -Scp

vlaaad10:07:53

that way you don’t touch your project’s deps.edn at all

p-himik10:07:15

Oh, it's now possible to use :deps in aliases. There's hope!

borkdude10:07:47

That overrides the base deps?

p-himik10:07:59

Yep, seems like it.

p-himik10:07:22

So basically I should be able to mimic an empty deps.edn. Will experiment later today.

borkdude10:07:54

note that you will still have clojure and spec on the cp by default

p-himik10:07:52

Hmm, right. But that should've really interfere with tools that are already meant to be run with clj.

borkdude10:07:31

I also use deps.edn for babashka projects, the extra deps don't hurt, but they can make searching the classpath a tad slower (like a few ms maybe)

p-himik10:07:55

In my case, they don't anything really bad as well, but I really want to get rid of those slf4j warnings.

Alex Miller (Clojure team)12:07:56

yes, using :deps / :paths in an alias is what we call "tooling" and will replace the project :deps/:paths (for the purpose of using a tool that shouldn't use the project classpath)

Alex Miller (Clojure team)12:07:43

until yesterday, you could only use those via -A, but as of the latest dev release yesterday, tooling -T is now a thing

p-himik12:07:02

Awesome, thanks!

Alex Miller (Clojure team)12:07:41

more to come on that and other changes in latest, just been kind of busy with ... other things :)

👍 3
Tamizhvendan S13:07:54

Hi, I am getting FileAlreadyExistsException while trying to build an uberjar Here is my deps.edn file

{:paths ["src" "resources"]
 :deps {io.appium/java-client {:mvn/version "7.3.0"}
        org.clojure/data.xml {:mvn/version "0.0.8"}
        metosin/reitit-ring       {:mvn/version "0.5.1"}
        metosin/reitit-middleware {:mvn/version "0.5.1"}
        metosin/reitit-spec       {:mvn/version "0.5.1"}
        mount                     {:mvn/version "0.1.16"}
        ring/ring-jetty-adapter   {:mvn/version "1.7.1"}
        org.slf4j/slf4j-simple    {:mvn/version "1.7.30"}}
 :aliases {:dev     {:extra-paths ["dev"]
                     :extra-deps  {org.clojure/tools.namespace {:mvn/version "1.0.0"}}}
           :uberjar {:extra-deps {seancorfield/depstar {:mvn/version "1.0.94"}}
                     :main-opts  ["-m" "hf.depstar.uberjar"]}}}

Tamizhvendan S13:07:49

Here is the command that I am running clj -A:uberjar my-app.jar

Tamizhvendan S13:07:17

Building uber jar: my-app.jar
Execution error (FileAlreadyExistsException) at com.sun.nio.zipfs.ZipFileSystem/createDirectory (ZipFileSystem.java:422).
META-INF/versions/9/

Tamizhvendan S13:07:19

Am I missing anything obvious here?

cap10morgan17:07:37

What are folks using to store / set their project's version number so it makes it into the generated pom.xml (in an automated way; think CD pipeline, not hand-editing the pom.xml)?

seancorfield17:07:23

I'm treating the pom.xml as the "system of record" for the version number and deriving it elsewhere from the POM file.

seancorfield17:07:16

(and that's the workflow that I use for all my deps.edn-based projects that get deployed to clojars -- note that Contrib libs such as core.cache, core.memoize, tools.cli that I maintain already follow this workflow with "Maven Release" process in CI editing the pom.xml to change the version number explicitly based on the version you enter in that Jenkins form)

cap10morgan19:07:22

@seancorfield OK, thanks. What tools do you use to read / bump it in the POM file? Curious if you have anything resembling the lein release workflow going there.

slimslenderslacks22:07:28

hey @cap10morgan, we're treating git tags as the source of truth for versions. When we deploy, the version in the generated pom.xml is made to be in sync with "git describe". Since the pom.xml is only ever needed for the deploy action, the pom.xml doesn't actually have to be checked in.

seancorfield22:07:34

Note that for http://clojars.org and also for http://cljdoc.org there's an expectation that pom.xml contains a lot more than just the dependencies and the version, so having it as a "loose" file outside git would be... ill-advised if the generation process doesn't include all that information (which lein does put there, but clojure -Spom does not, so I'm talking about deps.edn projects -- note that clj -A:new would generate initial pom.xml files with all the appropriate fields for http://clojars.org and http://cljdoc.org)

slimslenderslacks22:07:34

Ya, our process is still using lein for the pom.xml generation. That's super interesting @seancorfield - I was probably going to hit this in the near term as I had been planning on migrating to use clojure -Spom. Thanks!

seancorfield22:07:34

clojure -Spom is fine for updating <dependencies> in an existing pom.xml file but it only generates a very bare-bones version if no file exists. That's why I added a full-featured pom.xml to the templates that clj-new uses for app, lib, and template.

slimslenderslacks22:07:44

When git describe is the source of truth for version maybe we'd compose clojure -Spom to update the dependencies, with a git-describe->pom-version , and keep a clj-new generated pom.xml checked in.

slimslenderslacks23:07:47

It sounds like a checked in pom.xml might currently be the best place to version the extra metadata that clj-new adds.

seancorfield23:07:10

clj-new creates a baseline pom.xml. It never updates it after that.

👍 3
seancorfield23:07:52

clojure -Spom updates an existing pom.xml with the latest dependencies (from deps.edn) so as long as you always do it prior to a build/deploy cycle, I guess those changes don't really matter.

seancorfield23:07:23

Your git describe update also seems to be "do it before build/deploy" but don't really care about the changes.

seancorfield23:07:06

However, a lot of people would be very confused by a pom.xml file in a repo that didn't match the dependencies in deps.edn and/or didn't seem to reflect the current released version 🙂

seancorfield23:07:35

So whether you check it in or not after those changes probably depends on whether it's going into a public repo I suppose 🙂

slimslenderslacks23:07:24

Ya, I agree. It could be confusing to see a pom.xml with a weird version. You'd almost want the checked in copy to have a version of :project-uses-git-tags , and maybe have an empty dependencies, so it appears to be there just to define the extra maven stuff that you can't derive from elsewhere.

slimslenderslacks23:07:54

I don't mind having a file with the correct version, I was more just trying to avoid the commits that go along with those workflows.

slimslenderslacks23:07:32

Our unit tests and commit messages are the signals the produce Tags, and Tags are the signals that create new publishes to maven repos.

slimslenderslacks23:07:21

(from a CI/CD perspective)

seancorfield23:07:43

Yup. I think internal CI/CD processes are under different constraints and, unless you are publishing every successful build to (public) Maven Central / Clojars and also publishing every version to a (public) GitHub repo or similar, for internal releases you can definitely just keep a baseline pom.xml under version control and update that as needed for each build to your repo (and throw it away afterward).

Alex Miller (Clojure team)20:07:20

there are maven plugins for that although it's all pretty tedious

Alex Miller (Clojure team)20:07:05

the mvn release / deploy infrastructure does a lot for you but you have to buy pretty far into it all the way for it to be worth doing

Alex Miller (Clojure team)20:07:20

(but it's a lot less fragile than the lein release stuff)

cap10morgan20:07:53

Thanks Alex. lein release can indeed be fragile. 🙂

Alex Miller (Clojure team)20:07:44

the idea to put the version inside a file was probably bad from the beginning

seancorfield20:07:50

@cap10morgan For my clojars-deployed stuff, I just update it as part of my general release process where I'm updating the README and the CHANGELOG and I update both the <version> and the <tag> fields in the pom.xml file all at the same time.

seancorfield20:07:57

(so I grep for <old-version>|<new-version> so I can see all the places that refer to the current release and what will be the next one, and then I can visually verify that all references get changed)

seancorfield20:07:52

In most of my libs, that includes several places in the README and usually in the documentation files as well that refer to installation/getting started and when new features were introduced.

Alex Miller (Clojure team)20:07:06

I've automated that step on some libs :)

seancorfield20:07:07

I still have artisanal documentation 🙂