Fork me on GitHub
#tools-deps
<
2021-09-10
>
Ben Sless08:09:25

Not 100% if this should be asked here or in #tools-build as they probably have some overlap, can I use deps to build a project comprised of modules like https://github.com/metosin/reitit?

rickmoynihan09:09:21

IIRC juxt’s edge used to demonstrate this kind of setup: https://github.com/juxt/edge

rickmoynihan09:09:34

I assume it still does; but it constantly evolves

rickmoynihan09:09:20

See also @U04V70XH6’s blog posts on how to arrange a clj monorepo: https://corfield.org/blog/2021/02/23/deps-edn-monorepo/

Ben Sless09:09:26

I think the difference between Sean's use case and this one is how artifacts are built and distributed. When you have an internal monorepo for building deployable uberjars you don't really care about deploying your jars to some place like clojars, the builds are self contained. In this case I want to distribute the building blocks

Ben Sless09:09:26

Monorepo where the end product is deployable artifacts: I grok Monorepo where the end product is library modules with inter-dependencies: panik

rickmoynihan09:09:56

Fair enough — to be fair I’d like to do that too… Unfortunately some of our components are both open and closed source so it’s not really viable in a mono repo.

rickmoynihan09:09:05

Anyway I think tools-deps and tools-build or babashka tasks or a combination of them will give you much of what you need. Essentially all you need is a clj mono-repo, with a script that iterates over the modules and deploys them with depstar. Obviously assembling all that will involve building something somewhat bespoke… though you’d imagine it could be released as a clj build tool.

Ben Sless09:09:15

Yes. I found when I started exploring this problem space that dependencies specified as taken from a :local/root aren't taken into the pom, so I have to figure out something more clever

Alex Miller (Clojure team)12:09:32

Yeah, maven has no way to talk about that (or git deps)

Ben Sless12:09:45

Should they be resolved to an absolute path relative to the project root at pom creation time?

Alex Miller (Clojure team)13:09:01

the problem is the pom has no way to talk about "local" artifacts, only things in Maven repos

Ben Sless13:09:30

What I mean is, can local/root deps be resolved to where they will be once the artifacts are built and referred to as such in the pom? That way consumers of the module will be able to pull in the dependencies. Sort of like readlink or dereferencing

Alex Miller (Clojure team)13:09:57

"and referred to as such in the pom" -> no, there is no language for this in pom.xml

Alex Miller (Clojure team)13:09:44

pom.xml specifies deps only as groupId/artifactId/version for artifacts found in maven repos

Alex Miller (Clojure team)13:09:19

so you would need to at least "install" those artifacts to your local repo for them to be addressable by the pom.xml

Ben Sless13:09:07

I was thinking more along the lines of • I know at build time what the groupID/artifactID/version will be because I can get to the other repo by way of the local/root • Write those in the pom instead of install them locally • "eventually consistent" • ... • profit?

Ben Sless13:09:34

Stop me at any point if it seems unreasonable or like a bad idea

Alex Miller (Clojure team)13:09:26

is something deploying to a maven repo in this process?

Alex Miller (Clojure team)13:09:54

I mean, all of this seems like a bad idea if you're using Maven-oriented tools :)

Alex Miller (Clojure team)13:09:38

Maven does have a whole thing (multi-module projects, pom inheritance, etc) for building and deploying a set of related projects from a single repo

Alex Miller (Clojure team)13:09:28

but you need to be "doing Maven" for that whole thing to make sense

Ben Sless14:09:26

Brian Goetz likes to tell how people start questions with "why don't you just _" when they have a big ask, so I won't ask that, but I would say it would be nice to be able to do that, both for deployment (maven/clojars/artifactory) and for deps, to be able to pull in only parts of a code base which have already been organized as such Which reminds me of the spec-ulation talk, now

seancorfield16:09:54

@UK0810AQ2 Have you looked at how Polylith structures things? It completely separates the deployable part from the code so you could have a "project" for each thing you want to deploy to Maven/Clojars and separate projects for what your internally deployable stuff is -- and then the code is "assembled" from bases (usually things with a -main) and components (small "libraries").

seancorfield16:09:23

As noted in this thread, if you build something that you deploy to Maven/Clojars, then it must declare all of its dependencies as things that are also on Maven/Clojars -- but with a structure like Polylith, you can choose to build different artifacts for different audiences, so you could use git deps with :deps/root and :local/root dependencies and that would work "internally". Polylith's poly tool is a "project" inside the polyfy/polylith monorepo and can be used as a git dep, but they also build other artifacts from it.

Ben Sless16:09:59

Yes, but as you noted, it does not solve the dependencies issue, and I wouldn't want to maintain two dependency maps. I would like to programmatically transform the "is" to "ought". It might be easier for maven artifacts than for deps, actually

seancorfield16:09:40

Hmm, then I guess I don't understand what you're trying to do or why Polylith's approach won't solve your problem...

Ben Sless17:09:45

Just take reitit's structure as an example. Can it be done with deps?

seancorfield17:09:02

I don't use Leiningen so I don't really understand what's going on there -- but superficially it looks very, very similar to a Polylith structure with a workspace-level deps.edn (`project.clj` in reitit) and then each project/base/component having its own deps.edn (`project.clj` in reitit).

Ben Sless17:09:56

Yes, but building jars lets you refer to other components as dependencies in maven

dominicm17:09:48

@UK0810AQ2 Author of edge, so another person very familiar with this space. Doing the reitit thing with deps.edn is very easy! Let's start with our basic layout: Foo-core/ Foo-plugin/ Each folder will need: • deps.edn • pom.xml (to hold version, etc.) You can see that reitit updates the version everywhere on release: https://github.com/metosin/reitit/commit/8694d312f86d14ac7df3fa431af7cfba6873d8d8#diff-498cb648cb6ba613422c242cc068100c0bafa70b5cc71193570205abdac73b9e but you can maybe automate this with some tooling! deps.edn really has 2 modes of operation here: release & development. During development, you're looking to depend on the local copies of dependencies (presumably), and during release you're looking to depend on maven central versions of the same version as your version. One way to manage this would be to create a parent pom which has the version in and sets the versions for all the modules in your project. Another option is just to manually maintain which version your module depends on by doing it as part of the update to the pom.xmls in each module.

💯 2
dominicm17:09:39

Oh, so to do development in your deps.edn you should have a :dev alias like so:

:dev {:override-deps {Foo-core/Foo-core {:local/root "../Foo-core"}}}

Ben Sless17:09:31

Can children poms derive the version from the parent? Wondering how this will look like with tools build, too

seancorfield17:09:07

We use tools.build with Polylith so I'd say that's completely orthogonal.

dominicm17:09:40

tools.build doesn't interact here, particularly. @U04V70XH6 was working on stuff to make builds in other projects work, but that's an optimization that applies fractionally to libraries.

seancorfield17:09:51

There's certainly no conflict between having a unified :dev experience with everything using :local/root style deps and then building deployable artifacts that all have Maven-style coordinates and deps.

dominicm17:09:46

@UK0810AQ2 I believe a feature of maven called managed deps enables that, yes.

dominicm17:09:09

I think a question like this has come up before, and I'd like to do something like this for the juxt/yada repo if I get some time. Perhaps a small example repo would be worthwhile.

cap10morgan16:09:01

Is it expected behavior to get WARNING: Implicit use of clojure.main with options is deprecated, use -M when running clj --repl or clj -e '(+ 1 2)'? I don't necessarily have any aliases to use in those contexts.

hiredman16:09:03

% clj -M --repl
Clojure 1.10.2
user=>

cap10morgan16:09:58

ohhh... interesting. I didn't even realize / try it w/o an alias. hence the bracketed [aliases] in the help output. and the message literally tells me use -M facepalm😆 thanks @U0NCTKEV8!

hiredman16:09:03

I believe it is, the idea being if you want to invoke clojure.main you need to pass -M

👍 2
borkdude16:09:25

clj (without opts) still invokes a REPL though

👍 2
✔️ 2
Chase19:09:50

I'm trying to deploy to heroku but it seems they use Clojure 1.10.0.411 so they are not liking my up to date depstar invocation. I currently have this in my deps.edn :

:aliases {:uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.297"}}                           
                     :exec-fn hf.depstar/uberjar                                                                        
                     :exec-args {:aot true}}}
and was calling it with clojure -X:uberjar :jar my-app.jar but it doesn't like -X which I replaced with -A , :replace-deps had to be replaced with :extra-deps and I'm at a loss with what to do for the :exec-fn ... . What would you advise there?

Chase14:09:04

Oh man, I wish I had found this earlier yesterday. That $PORT thing was driving me crazy before I discovered what I was doing wrong.

seancorfield19:09:58

Petition Heroku to stop using such an ancient version of the CLI? 🙂 I'm a bit surprised you can't specify which CLI version you want, like you can for CircleCI or GitHub Actions...?

seancorfield19:09:29

Given that, you'll need to switch to the old -M invocation of depstar (which is no longer documented!)...

seancorfield19:09:20

You'll want :main-opts in :uberjar with ["-m" "hf.depstar.uberjar" "-C"] plus whatever additional args you need @chase-lambert

seancorfield19:09:08

The old -main entry point doesn't support all the options that the -X entry point does. Here's the excerpt that does the argument parsing:

("-C" "--compile")   [{:aot true} (next args)]
                ("-P" "--classpath") [{:classpath (fnext args)} (nnext args)]
                ("-D" "--debug-clash") [{:debug-clash true} (next args)]
                ("-h" "--help")      [{:help true} (next args)]
                ("-J" "--jar")       [{:jar (fnext args)} (nnext args)]
                ("-m" "--main")      [{:main-class (fnext args)} (nnext args)]
                ("-n" "--no-pom")    [{:no-pom true} (next args)]
                ("-S" "--suppress-clash") [{:debug-clash false} (next args)]
                ("-v" "--verbose")   [{:verbose true} (next args)]
                ("-X" "--exclude")   [{:exclude (fnext args)} (nnext args)]
so you'll probably want --jar my-app.jar -m your-main-ns maybe others?

Chase19:09:54

Awesome, thanks for this.

Chase19:09:47

Yeah I wish the official heroku buildpack was more up to date. It seems they are closing the issues asking for an official deps.edn version. There are various workarounds and community built buildpacks floating out there

seancorfield19:09:48

According to Heroku's own docs you can specify a more modern CLI via the CLOJURE_CLI_VERSION environment variable.

Chase19:09:14

well then. I did not see that solution in the clojure buildpack github repo

Chase19:09:39

Awesome, I'll just find a way to set that variable then so I don't get future conflicts as I continue to add to my deps.edn

seancorfield19:09:22

Here's the lines in their build pack that handles arbitrary CLI installation: https://github.com/heroku/heroku-buildpack-clojure/blob/main/bin/compile#L33-L37

seancorfield19:09:00

Settting "config vars" (aka environment variables I assume): https://devcenter.heroku.com/articles/config-vars#managing-config-vars

seancorfield19:09:31

(I'm guessing here, based on their docs -- I have never used Heroku)

Chase19:09:29

That worked! Thanks so much. I feel silly for not checking that option out first

seancorfield20:09:45

NP. By learning a bit more about Heroku, I'm better able to help others who may have Qs about it, even if I don't use it myself 🙂

practicalli-john08:09:39

I used to work at Heroku, so am fairly familiar with there tooling. The buildpack is something I'd like to revisit as my current approach is not as smooth as it should be. If there isn't a suitable Heroku buildpack, then I'll consider creating a suitable community version.

2
Mark Wardle20:09:04

Hello all. Happy user of deps.edn and depstar to build jars and uberjars - should I be thinking to replace my depstar usage with a tools.build build.edn? Any recommendations - I don’t do anything very complex.

seancorfield20:09:21

@mark354 For library JARs, I have switched completely to tools.build -- take a look in projects like next.jdbc or honeysql to see the sort of stuff I'm doing with tools.build nowadays.

seancorfield20:09:59

Even depstar's own CI pipeline uses tools.build to run tests and build its own JAR! 🙂

👍 2
seancorfield20:09:59

For most folks, using tools.build for uberjars is going to work but there are currently some gaps between uber in tools.build and what depstar does so right now I still recommend depstar for building uberjars.

Mark Wardle20:09:23

That is very helpful thank you - I will keep watching progress! Thank you.

seancorfield20:09:16

However, I'm actively talking with Alex about this and we've been discussing a path forward to help get uber to parity with depstar's uberjar functionality -- mostly around how the latter handles OSS license files in libraries that are pulled into the uberjar and also the thorny problem of log4j2 plugins cache files. My goal is to work with Alex to get tools.build to a point where I can retire depstar completely and recommend everyone use tools.build instead.

👍 2
seancorfield20:09:49

(also, there's a #tools-build channel for anyone who wants to dig deeper into that stuff)

👍 4