Fork me on GitHub
#leiningen
<
2018-10-29
>
pmonks17:10:02

So after a lot more experimentation with classifiers, dynamic version strings, etc., it looks like the only way I'm going to be able to achieve what I'm after (single project.clj that can emit two different, JVM-specific artifacts that have different upstream dependencies, depending on which JVM the build is run on) is by twizzling the project name component in the defproject form. e.g.

(def project-name (... logic that calculates a project name based on the JDK we're running on ...))

(defproject project-name "0.1.0-SNAPSHOT" ...)
But I've been unable to get that to work. Any tips, tricks or pointers?

andy.fingerhut18:10:35

Maybe you can write a macro that takes the project-name as arg, and returns the defproject form you want?

andy.fingerhut18:10:06

although that may run into problems with the fact that you want to calculate the project name at run time, not compile time.

andy.fingerhut18:10:16

Possibly-effective hack: shell/clj/Python/etc. script that writes the project.clj file and does build/deploy/whatever commands, then tweaks the project.clj file and repeats.

mikerod18:10:41

@pmonks I thought you were going with a version string change? at this point, the best I can think of is similar to what @andy.fingerhut mentioned concerning having an external script that picks the different jvm’s for you to run lein with, so a defproject setup similar to this

mikerod18:10:31

So it is flexible to adapt to different jvm versions

pmonks18:10:46

@andy.fingerhut yeah that thought crossed my mind too, but it felt to me like a bridge too far - like I'm definitely headed down a bad path. 😉

pmonks18:10:28

@mikerod it turns out that when you use a version number suffix, the Maven machinery still only has a single POM file, and I need two (recall that my upstream dependencies differ, depending on which JVM is in use).

mikerod18:10:31

The a script could run it like

# jvm x 
LEIN_JAVA_CMD=jvm-x-path lein <task>

# jvm y
LEIN_JAVA_CMD=jvm-y-path lein <task>

mikerod18:10:54

yeah, at this point though, I’d go with the version appraoch

pmonks18:10:55

Sure, but I can't make assumptions about which JVM(s) consumers of the library are using.

mikerod19:10:01

and just manage the artifacts via an external script

pmonks19:10:08

This is a general purpose library that (hopefully!) many other folks will consume and enjoy. 😉

mikerod19:10:23

you have conditional deps depending on jvm?

pmonks19:10:24

So I can't just assume they have the same JVMs installed that I do, let alone in the same locations.

mikerod19:10:26

so you have to need to know

mikerod19:10:41

the script is for you to deploy things

pmonks19:10:44

Basically it's all fallout from Oracle's "project jigsaw". And it's a huge pain that I'd rather not have to deal with...

mikerod19:10:50

a consumer wouldn’t write anything, it’d just build with whatevr jvm they have lein using

mikerod19:10:06

not sure why a consumer would be building your project.clj though, but they could

pmonks19:10:11

They wouldn't.

pmonks19:10:36

But they do need to select which "edition" of my library they wish to consume, to ensure they get the correct upstream dependencies.

mikerod19:10:54

So yeah, you use this, and deploy artifacts of the form:

org.mine/some-project 0.1.0-SNAPSHOT-jdk8
org.mine/some-project 0.1.0-SNAPSHOT-jdk10

pmonks19:10:03

You can, but those share a pom.

pmonks19:10:09

I tested that first, as we discussed last week.

mikerod19:10:11

no, built entirely separately

pmonks19:10:21

Nope that's not how Maven / Clojars work.

mikerod19:10:22

2 different invocations of lein

pmonks19:10:32

The second build overwrites the pom of the first.

mikerod19:10:42

oh, you are concerned with how to deploy multiple artifacts then

mikerod19:10:52

these are classified, could deploy both together and would have classifiers

pmonks19:10:07

I can't use classifiers, since I need to build on different JVMs.

pmonks19:10:17

And leiningen can't "split" classifiers to run on different JVMs.

mikerod19:10:20

but that doesn’t align with what you showed from google doing a similarly hacky classifier deployment based on versions

mikerod19:10:34

again, lein runs 2 times

mikerod19:10:38

external process

pmonks19:10:43

Recall that my dependencies are either included in the JVM (for JVM versions 1.7u6 through 10) or need to be explicitly pulled down (JVM v11+).

pmonks19:10:53

So I need two poms./

mikerod19:10:58

2 times

LEIN_JAVA_CMD=jvm-x-path lein <task>
LEIN_JAVA_CMD=jvm-y-path lein <task>

mikerod19:10:04

each has own everything

mikerod19:10:18

completely compile-time built-in due to the eval-syntax of the defproject based on lein jvm

pmonks19:10:42

At at the risk of sounding like a broken record, upon deployment to clojars, LEIN_JAVA_CMD=jvm-y-path lein <task> overwrites the POM file that was previously generated by LEIN_JAVA_CMD=jvm-x-path lein <task>.

mikerod19:10:43

my snippet above shows how at compile-time the jvm-specific-deps are chosen

pmonks19:10:57

So I end up with 1 pom and 2 jars, which is exactly the opposite of what I need.

mikerod19:10:00

it depends what you are deploying

mikerod19:10:08

if you give them different versions - they aren’t overwriting

pmonks19:10:14

I confirmed it.

pmonks19:10:20

I thought as you did, until I tried it out.

pmonks19:10:22

And it didn't work.

mikerod19:10:23

if you use classifiers - they need to deploy at the same time, because you can’t “overwrite” releases

mikerod19:10:28

(in may repo setups)

pmonks19:10:31

I can't use classifiers.

pmonks19:10:45

Each pom has to be built on a different JVM version, or else the unit tests won't pass.

andy.fingerhut19:10:52

If you build one JAR, overwriting pom file, and deploy (in the script), then start over with a different JVM version, it shouldn't matter that the pom file gets overwritten after the deploy?

mikerod19:10:09

if you call one 1.0-jdk8 and the other 1.0-jdk9 in the <version> of the pom

pmonks19:10:10

The poms are different.

mikerod19:10:19

you deploy one completely before the 2nd

mikerod19:10:29

you showed me that google ended up with a mvn repo structure like

mikerod19:10:36

1.0-jdk8/stuff

pmonks19:10:40

@mikerod I did exactly that. Perhaps I'm not explaining myself very clearly, but Maven / Clojars elide the "qualifier" on the end of the version string.

mikerod19:10:48

if that is the case, it wouldn’t override

pmonks19:10:53

So you end up with two poms that "land" in the same directory + file on Clojars.

andy.fingerhut19:10:00

If you can find any way to do what you want via a sequence of commands from an interactive shell, that produces the correct results, script that.

mikerod19:10:01

if it deployed as a real classifier then you couldn’t deploy 2 separate times to the same coordinate

pmonks19:10:03

i.e. the second overwrites the first.

pmonks19:10:11

@mikerod I can't use classifiers.

mikerod19:10:30

the maven definition of a classifier has some vague semantics around final repo structure

pmonks19:10:40

I need each "build" to use a specific JVM version, and classifiers can't be specified to use a particular JVM.

pmonks19:10:03

(since leiningen itself is running within a JVM - it can't swap that out at runtime).

mikerod19:10:06

you showed me some google repo, that used a “classifier version” structure, but it was actually imposed on the directory structure itself - so it sort of “wasn’t a classifier”, but could be used like one as a mvn coordinate

mikerod19:10:22

you have full control on lein’s jvm

mikerod19:10:23

as i showed

pmonks19:10:24

Correct - to get that out of a single project.clj I need to change the project name.

mikerod19:10:29

LEIN_JAVA_CMD=jvm-x-path lein <task>

mikerod19:10:31

that uses jvm x

pmonks19:10:32

Hence my question this morning.

mikerod19:10:43

you could compile-time hard-wire your defproject do about anything based on that

pmonks19:10:45

If the project name were a string (as the version is) this would be trivial.

pmonks19:10:51

But it's not - it's a symbol.

mikerod19:10:56

no reason to mess with project name…

mikerod19:10:12

but I may give up here, since not getting a point through

mikerod19:10:17

and don’t know the google example you gave before anymore

pmonks19:10:18

Yep - that's the only way to influence how Leiningen's underlying Maven machinery determines where to write files during deployment.

andy.fingerhut19:10:30

Do you have a manual sequence of steps you know how to follow that produces the results you want?

mikerod19:10:37

or if you don’t care about version that much, could just pick a naming scheme that doesn’t cause it to “look like a classifier”

pmonks19:10:38

Outside of leiningen, you mean?

andy.fingerhut19:10:40

even if that sequence of steps involves hand-editing the project.clj and/or other files?

pmonks19:10:53

Let me pseudo code it...

andy.fingerhut19:10:14

Make a script that does those steps, with instead of "hand-edting one or more files" does sed or any other program to edit those files that you can think of.

pmonks19:10:05

1. Select Oracle JVM v1.8 (there are a million ways to do do this, all of them unpleasant) 2. Run unit tests, without the additional dependencies (`lein test`) 3. Build JAR and deploy, without the additional dependencies (`lein deploy clojars`), ensuring the deployed artifacts (BOTH JAR AND POM) have unique GAV coordinates 4. Select Open JDK v11 (there are a million ways to do do this, all of them unpleasant) 5. Run unit tests, with additional dependencies (`lein with-profile +openjfx11 test`) 6. Build JAR and deploy, with additional dependencies (`lein with-profile +openjfx11 deploy clojars`), ensuring the deployed artifacts (BOTH JAR AND POM) have unique GAV coordinates

andy.fingerhut19:10:10

Could be bash, could be Python, clj, etc. Whatever you are most familiar and comfortable with maintaining.

andy.fingerhut19:10:04

I don't even need to see what the other steps are, literally. If you can write them as commands in a bash script, Bob's your uncle.

andy.fingerhut19:10:22

If there are any steps that you don't know how to write as a command in a bash script, please ask -- we can probably figure out a way to do it.

pmonks19:10:31

What about my fellow library developers on Windows?

pmonks19:10:49

Sorry - that was a mostly rhetorical question.

mikerod19:10:51

@pmonks who all is building this?

mikerod19:10:05

if you need windows support also, for building, then yes, things get more complicated with scripts hah

pmonks19:10:06

Yes scripting is always a back door way out. But it's a pretty sucky one imho.

mikerod19:10:12

I’ll probably bow-out of trying to fix this issue, since I don’t feel like I’m adding value at this point 😛

mikerod19:10:22

but you can hackily change project name for what that’s worth to you

mikerod19:10:31

(defproject #=(symbol "foo") ....)

mikerod19:10:45

so you could conditional logic, based on jvm or something, the project name if you wanted to

pmonks19:10:31

@mikerod that doesn't work:

(def project-name "foo")

(defproject #=(symbol project-name) "0.1.0-SNAPSHOT"
$ lein jar
Created [snip]/target/project-name-0.1.0-SNAPSHOT.jar

pmonks19:10:18

(and yes I guess I could put a mound of logic inside the call to symbol, but that gets pretty messy)

mikerod19:10:59

if you call it right

mikerod19:10:15

just call a fn that calculates thing in #=

mikerod19:10:01

don’t try to refer to symbols within the #= literal

mikerod19:10:06

that is just quirks around #=

mikerod19:10:48

(defn project-name [] (symbol "foo"))

(defproject #=(project-name) "0.1.0-SNAPSHOT" ...)

👍 1
mikerod21:10:06

@pmonks you tried :leaky metadata on the profile too?

mikerod21:10:18

just checking, otherwise can be looked into, good to have the lein issue logged

mikerod21:10:13

I believe if you mark your profile :leaky it will work

mikerod21:10:31

but that is sort of counterintuitive to me how with-profile can be ignored when not :leaky. That may be by design, not sure.

pmonks18:10:02

^:leaky was the missing piece - thanks @mikerod !

mikerod19:10:19

I think lein v3 is even opting to always only go with #= reader dispatch macro instead of the classically used ~, but I have no more details on that. It’s just in a comment I see in the source - https://github.com/technomancy/leiningen/blob/7357010431c926abb559d55b295fe1ba7791ed20/leiningen-core/src/leiningen/core/project.clj#L198

andy.fingerhut19:10:44

You could write a PowerShell script if you were so inclined (or had a colleague who is)

andy.fingerhut19:10:19

At times, I consider scripting the unequivocal front door 🙂

andy.fingerhut19:10:53

There's also LSW

pmonks19:10:08

Going off on a tangent for a moment, are there any thoughts around revisiting build tools, especially in light of clojure/tools.deps?

mikerod20:10:43

I think the idea of tools.deps is cool that it tried to focus on one small part of what build tools bundle in most of the time - dep mgmt

mikerod20:10:05

however, eventually you have to compose a bunch of parts together either yourself via scripts, or a build tool that tries to be all in one

mikerod20:10:35

I’m not sure where the tools.deps idea is headed beyond tools.deps itself, but already, looks to be a script-centric view of doing things when I read how people are using it in their stack

mikerod20:10:49

I think integrating it with lein and removing that “classpath” part is at least cool/interesting

mikerod20:10:27

in terms of revisiting build tools overall, when it comes to complicated build and deployment setup, I’m not sure there is an easy answer for build tools. It seems that many exists and everyone is always unhappy with them all eventually 😛

mikerod20:10:45

or we just end up scripting a bunch of steps together from separate tools

andy.fingerhut19:10:30

Definitely -- I see it discussed on the #tools-deps channel, and someone there has been publishing and perhaps maintaining a list of tools.deps based tools

andy.fingerhut19:10:07

I haven't used them myself, so can't help you with what exists vs. what hasn't been developed yet.

pmonks19:10:25

FWIW I've played with the lein-tools-deps plugin, and it's fine as far as it goes. The issue comes in that most other leiningen plugins can't use deps expressed in deps.edn.

pmonks19:10:51

So stalwarts like lein-ancient become useless, which is a big loss.

pmonks19:10:09

I guess I'm just wondering if the consolidation of dependency management by the core Clojure team will sufficiently break the existing build tools that it's a logical point to re-evaluate build tools more generally...

andy.fingerhut19:10:54

By 'break existing build tools' you mean because some project you want to build via Leiningen doesn't come with a project.clj file? I don't understand "break" there.

pmonks19:10:03

I mean "break" in the sense of "these tools can't directly consume deps.edn files yet, and it seems like a substantial effort to add support for that and deprecate their own native dependency management mechanisms (given the ripple effect through the plugin ecosystem etc.)".

andy.fingerhut19:10:19

Sure, but it is "opt-in" breakage, e.g. moving from major version 1 to version 2 on your own time frame (or not)

andy.fingerhut19:10:41

Leiningen-specific plans for supporting tools.deps seems best discussed here, but at least lein ancient has a tools.deps-based replacement someone has already developed: https://github.com/RickMoynihan/lein-tools-deps/issues/63

👍 1