Fork me on GitHub
#leiningen
<
2018-10-22
>
deactivateduser20:10:51

G'day folks! I'm trying to configure a project to deploy two JVM-version-specific artifacts for the same version, but I'm running headlong into my shallow Leiningen knowledge. Apologies in advance for the following Wall-O-Text™. Background: * The project has different dependencies, depending on which JVM it's on - I currently handle these via :profiles in the project.clj [1]. Objectives: 1. To deploy two "editions" of the library's JAR and POM (since the dependencies are different) to Clojars for each release. 2. To make it as easy as possible for consumers to select which "edition" of the library they want to use (e.g. via a :classifier [2] - that seems like an elegant way to handle this). Challenges: 1. Each of the JVM-version-specific artifacts has to be built on the JVM it's targeting - this is required so that the CI environment can run unit tests first, prior to deployment (the tests won't pass if they're run on the wrong JVM). 2. This implies that there will need to be two independent deployments to Clojars, for the same version of the library (albeit with different classifiers or whatever). I haven't been able to find out whether Clojars allows this kind of thing or not - I know it doesn't normally allow redeployment of non-SNAPSHOT versions though, which makes me pessimistic. Learnings to date (take with a grain of salt - may be wrong), and some questions: 1. At least some Java libraries achieve this by twizzling their version strings directly (e.g. Google Guava [3][4]). Does Leiningen allow a project's version number to be overridden elsewhere in a defproject form? 2. :classifiers don't seem to be overridable in a :profile (bug?), which makes me wonder if they're the right tool for this job. Or perhaps :profiles is the wrong tool for this job? 3. The pom task doesn't seem to take :classifiers into account - it always generates an unversioned pom.xml in the root of the project. Unless the deploy task does some additional magic, this seems like it'll be problematic (i.e. cause clashes at deployment time). 4. The :jar-name [5] property gets me halfway there (let's me set the JAR name on a profile-by-profile basis), but again I need both the JAR and the POM to be versioned, due to the dependency differences. Any thoughts / suggestions? [1] https://github.com/fn-fx/fn-fx/blob/master/project.clj [2] https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L504-L510 [3] https://repo1.maven.org/maven2/com/google/guava/guava/ [4] https://github.com/google/guava/blob/master/android/pom.xml#L14 [5] https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L405-L407

Alex Miller (Clojure team)20:10:23

Clojars shouldn’t (hopefully) have any problem with this - as a repo handling standard Maven repository format, this is not unusual. All of this is supportable in a Maven pom, usually with the assembly plugin (FYI). Not sure about any of the lein-specific stuff.

mikerod21:10:49

@deactivateduser10790 > 2. This implies that there will need to be two independent deployments to Clojars, for the same version of the library (albeit with different classifiers or whatever). I haven’t been able to find out whether Clojars allows this kind of thing or not - I know it doesn’t normally allow redeployment of non-SNAPSHOT versions though, which makes me pessimistic. why does it imply that though?

mikerod21:10:24

Could you not create each artifact with their respective jvm version, placing both into a :target-dir and then do a deploy of all those things after the fact?

deactivateduser21:10:31

Because each artifact comes from a different JVM and Leiningen can’t easily switch JVMs mid stream.

deactivateduser21:10:48

Sure, but this doesn’t let me run the build with one JVM, then switch to a different JVM and run the build again, then finally combine the results (on either JVM) and deploy.

mikerod21:10:28

So say you specified a different :java-cmd in 2 separate profiles

deactivateduser21:10:20

How does each artifact’s version string then get updated with the right classifier, given I’ve found the :classifier doesn’t get overridden within a profile?

mikerod21:10:48

and had 2 :aliases

:aliases {"jdk7stuff" ["do"
                       ["with-profiles" "jdk7setup" <etc>]]
          "jdk8stuff" ["do"
                       ["with-profiles" "jdk8setup" <etc>]]}
can’t do that?

mikerod21:10:19

oh, you are finding an issue with classifier behavior?

deactivateduser21:10:29

Also, that breaks Travis’ (nice!) per-JVM job parallelisation. Not a showstopper mind you, but a loss nonetheless.

mikerod21:10:05

I definitely can’t speak to Travis

mikerod21:10:10

but think lein could do it

mikerod21:10:19

except your classifier problem, but I’m not aware of the issue you are referring to there

mikerod21:10:31

you can sort of hack around that in some ways

deactivateduser21:10:37

From my original post: > 2. :classifiers don't seem to be overridable in a :profile (bug?), which makes me wonder if they're the right tool for this job. Or perhaps :profiles is the wrong tool for this job?

mikerod21:10:54

but just making a target artifact name that reflects the classifier in it, but not sure that is great

mikerod21:10:12

yeah, sorry, overlooked the part where you said that

deactivateduser21:10:13

Yeah I tried that, but can’t get two POMs out the backend.

mikerod21:10:55

classifiers build like a profile right?

deactivateduser21:10:56

The pom task doesn’t seem to respect :classifiers, and I haven’t found a way to tell it to use a different name (or directory).

mikerod21:10:06

can’t put the :java-cmd profiles in those?

mikerod21:10:54

classifiers wouldn’t affect a pom right?

deactivateduser21:10:57

I haven’t tried that, mostly because the docs for classifiers are so weak I didn’t even think of it. I’ll give it a try now.

deactivateduser21:10:49

They should - after all the Maven concept of a classifier is that it’s part of the version.

deactivateduser21:10:19

So different “versions” (different only in their classifier component) should be able to have different poms.

deactivateduser21:10:32

And I need that, as the dependencies are different between the two artifacts.

mikerod21:10:06

I didn’t think that is how classifiers work

mikerod21:10:12

could be wrong, but thought they had same pom

mikerod21:10:15

different file names

mikerod21:10:31

when referred to as a dependency, then you include the classifier

mikerod21:10:35

I’m likely wrong

mikerod21:10:07

well, not seeing it in the xml for a pom

deactivateduser21:10:57

It's external to the POM

deactivateduser21:10:05

It's handled via the directory structure. See Google Guava for an example: https://repo1.maven.org/maven2/com/google/guava/guava/

deactivateduser21:10:33

(albeit they achieve their "jre" vs "android" editions by messing with the version numbers directly, possibly because of some of the same issues I'm facing)

deactivateduser21:10:10

At the end of the day, I just want to deploy something just like Google Guava, from a single project.

deactivateduser21:10:28

One project, two emitted artifacts, ideally differentiated by something in the version number.

deactivateduser21:10:00

Interestingly, for now the JAR files themselves are in fact identical - it's only the POMs that are different. But that may change down the line.

mikerod21:10:07

ok, so I already do deploy a jar and standalone classfiier jar

mikerod21:10:10

in one push via deploy

mikerod21:10:20

and the difference is indeed in the artifact name - not pom

mikerod21:10:41

so if the pom isn’t different between the two, not sure why you think the pom task should respect the classifier at all - doesn’t affect it

mikerod21:10:48

(I get that guava may be doing things extra)

deactivateduser21:10:50

The pom IS different.

mikerod21:10:52

but in general

deactivateduser21:10:09

In fact that (for now) is the only difference between my two "editions" of the artifact.

deactivateduser21:10:17

(the code is identical, the dependencies are not)

mikerod21:10:31

so you say, a classified artifact in maven

mikerod21:10:36

must have the classifier explicitly in the pom

mikerod21:10:11

it doesn’t seem that maven cares about that at least

deactivateduser21:10:19

Forget classifiers for a second - they're potentially part of the solution, but don't help explain the problem, and I don't know if I've expressed the problem very well yet. 😉

mikerod21:10:21

because I deploy artifacts with 100% the same pom

mikerod21:10:33

but different artifact names - and can bring them in via aether as dependencies

mikerod21:10:41

with :classifier specified individually

mikerod21:10:59

sure, just trying to weed through the several problems we have here hah

deactivateduser21:10:54

Forget also the consumption-side use of :classifier - I'd like to be able to offer that UX to consumers, but if it makes the production side too hard then I'm willing to ditch it for equivalently easy methods (e.g. proprietary version strings, or even different artifact ids, if need be).

deactivateduser21:10:11

My issue right now is that I can't even produce the two artifacts I need.

deactivateduser21:10:22

And consumption-side UX is moot until I get to that point. 😉

mikerod21:10:30

My theory is that you may be able to get multiple lein tasks to run, that produce artifacts to the same :target-dir and each having a name that represents its classifier. Once there, you could do a deploy after those tasks are done with their own Jvms. aether (which lein uses), does allow you to deploy multiple artifacts and will respect classifiers.

mikerod21:10:41

yeah, not sure you’ll love the solutions to it if you are wanting something super dead simple

deactivateduser21:10:55

I'm ok with complex on the production side. Less so on the consumption side.

mikerod21:10:57

I think you are going to have to run multiple lein tasks that output to perhaps separate directories

mikerod21:10:06

and then have something to push them both into a final directory and deploy all

deactivateduser21:10:51

I might as well just use bash for all of that. I suspect leiningen + plugins will only make that difficult.

deactivateduser21:10:12

(which is also what I was hoping to avoid, since it causes friction with TravisCI)

mikerod21:10:02

I use lein-shell sometimes for things

mikerod21:10:10

if it is some file movement things that are hard to pull off

mikerod21:10:17

or you can write a lein plugin that specifically deals with situations

mikerod21:10:28

but yeah, I’m not sure how this interacts with some Travis features you are wanting

mikerod21:10:45

one problem is going to be that jar tasks cleans the :target-dir

mikerod21:10:49

autocleans

mikerod21:10:57

I’m not sure if you are building uberjars to or what

deactivateduser21:10:03

Yeah I noticed that. Kind of a pain.

mikerod21:10:04

but if you were just making 2 jars

mikerod21:10:11

you could probably turn autoclean off

mikerod21:10:23

and then get them both to appear in same :target-dir one after the other

deactivateduser21:10:25

Yeah no uberjars here - just regular JARs and their (different!) POMs.

deactivateduser21:10:36

That doesn't solve the POM collision though.

deactivateduser21:10:04

I still need to have two POMs and two JARs, and haven't found a way to get the pom and/or deploy plugins to do that.

mikerod21:10:12

I don’t know how to get your pom’s different, but I’m just assuming you can get :classifiers to write the artifact names correctly so that it deploys with the correct classified name

mikerod21:10:18

2 jars, maybe

lein do jdyXstuff, jdkYstuff
If you turn :auto-clean false

mikerod21:10:25

have neither of those do a deploy

mikerod21:10:17

I don’t know a great way of elegantly specifying files yourself directly to the deploy tasks though

mikerod21:10:28

because it takes more args that you’d be repeating from the project, maybe works to do it though

deactivateduser21:10:33

Yeah though now I'm back to the "I need to switch JVMs between classifiers" problem.

mikerod21:10:53

each lein task has its own :java-cmd

deactivateduser21:10:55

Sadly this code won't work if the dependencies & JVM versions don't exactly match.

mikerod21:10:04

that comes from a profile, link the profiles to the classifier in :classifiers

mikerod21:10:15

:profiles {:jdk-x-stuff {:java-cmd "jdk-x"}
           :jdk-y-stuff {:java-cmd "jdk-y"}}
:classifiers {"jdk-x" :jdk-x-stuff
              "jdk-y" :jdk-y-stuff}

deactivateduser21:10:03

Right, but that doesn't generate two separate poms.

mikerod21:10:11

the pom thing sounds irrelevnat

mikerod21:10:16

why do you think the pom has to differ?

mikerod21:10:19

who is that helping?

deactivateduser21:10:20

Recall that the dependencies are different

mikerod21:10:31

yes, the artifacts will be named differently

mikerod21:10:46

your-project-1.0-jdk-x.jar
your-project-1.0-jdk-y.jar

deactivateduser21:10:49

On JVM 1.8 the classes my library uses are part of the JDK. On JVM 11+ they're provided by a separate library.

mikerod21:10:50

2 different Maven coords

deactivateduser21:10:09

Which I need to include in my library's dependencies, or else JVM 11 users of my library will be rather sad.

mikerod21:10:10

[your-project "1.0" :classifier "jdk-x"]
[your-project "1.0" :classifier "jdk-y"]

mikerod21:10:23

finally clicked to me what you are saying

mikerod21:10:27

you have different deps for each

deactivateduser21:10:28

How do I generate two poms, one for jdk-x and one for jdk-y?

deactivateduser21:10:45

MY library has "variable" dependencies, depending on JVM version.

mikerod21:10:58

hmm are classifiers the wrong tool for that job I wonder

mikerod21:10:12

that’s interesting to me

deactivateduser21:10:17

So I want to ship / emit / deploy two different "editions" of my library, ideally with the same version number, but with different classifiers (or whatever), so consumers can easily choose between them.

mikerod21:10:19

because mvn typically drops a pom.xml file in the repo

mikerod21:10:25

for all classified artifacts

deactivateduser21:10:29

Right, and you can see how Google Guava does it.

deactivateduser21:10:33

They "stuff" the version number.

mikerod21:10:35

so it’s weird when there is no unified pom to put there

deactivateduser21:10:46

Same version number, but different classifiers for JRE vs Android.

mikerod21:10:46

yeah, if you stuff the version, it is a different thing basically

deactivateduser21:10:57

Not really - Maven just thinks it's a classifier (since those have a fixed position in the version number, but other than that are just a freeform string value).

mikerod21:10:09

trick the naming thing I guess

mikerod21:10:12

I guess that’s legal

mikerod21:10:49

you had an issue trying to get lein to take a version from a profile?

deactivateduser21:10:52

Right, but I need to twizzle the version # in order to get directories.

deactivateduser21:10:05

Naming isn't enough.

deactivateduser21:10:28

These things have to go into different directories (again look at how Google Guava does it: https://repo1.maven.org/maven2/com/google/guava/guava/ ),.

mikerod21:10:57

yeah, that’s why it seems like a hack to me

deactivateduser22:10:04

For every "version" there are two directories: 1. major.minor-jre 2. major.minor-android

mikerod22:10:04

it’s not really a classifier, it’s really a version

mikerod22:10:07

at least in the mvn file structure

deactivateduser22:10:23

Actually Maven will consider "jre" and "android" to be a classifier.

mikerod22:10:29

it doesn’t match the explanation from the mvn docs on what a classifier is

deactivateduser22:10:36

Those can be anything you want (with a few "special" values that have meaning to Maven for the purposes of version sorting).

mikerod22:10:38

yeah, but I think it’s a hack way to do it

mikerod22:10:46

like it isn’t strictly following what maven does for classifiers

deactivateduser22:10:52

Again not really - that's why Maven has classifiers.

mikerod22:10:12

> classifier: The classifier distinguishes artifacts that were built from the same POM but differ in content. It is some optional and arbitrary string that - if present - is appended to the artifact name just after the version number. As a motivation for this element, consider for example a project that offers an artifact targeting JRE 1.5 but at the same time also an artifact that still supports JRE 1.4. The first artifact could be equipped with the classifier jdk15 and the second one with jdk14 such that clients can choose which one to use. Another common use case for classifiers is to attach secondary artifacts to the project’s main artifact. If you browse the Maven central repository, you will notice that the classifiers sources and javadoc are used to deploy the project source code and API docs along with the packaged class files.

deactivateduser22:10:13

To give an escape hatch for libraries that have a single logical version with multiple different artifacts.

mikerod22:10:44

I get that it is a way to do multi-artifact from a pom

mikerod22:10:53

but putting it in sep version directories is giving it multiple poms

deactivateduser22:10:59

Right. The JDK14 / JDK15 example is a good one, if you imagine that a 3rd party dependency is needed to support JDK14, but not JDK15 (or vice versa),.

mikerod22:10:04

I think the naming convention is just ambiguous

deactivateduser22:10:07

That's basically exactly my situation.

mikerod22:10:15

I get your situationa nd why google did it

mikerod22:10:22

still doesn’t match the notion of a classifier from taht paragraph

mikerod22:10:29

" The classifier distinguishes artifacts that were built from the same POM but differ in content.”

deactivateduser22:10:42

Right. And what I figured out is that they mess with the version number during the build, so effectively they provide multiple POMs.

mikerod22:10:44

different directories with different dependencies - not the same pom

deactivateduser22:10:57

Question is - does Leiningen allow the version number to be messed with, ideally inside a profile.

deactivateduser22:10:03

Cause then my problem would be solved. 😉

mikerod22:10:03

well, I guess I’m wrong somewhat, you can do different bulid steps with deps in the same pom, so nvm

mikerod22:10:18

yeah, I know that’s what your question is coming down to, and it’s probably doable

mikerod22:10:29

so problem solved, it’s probably doable

deactivateduser22:10:42

I see bash in my immediate future...

deactivateduser22:10:45

And TravisCI pain...

deactivateduser22:10:06

Thanks for talking this through with me though - really appreciate it!

deactivateduser22:10:14

Glad I'm not just missing something obvious.

mikerod22:10:29

yeah, sorry for drilling with questions

mikerod22:10:38

just sort of confusing and can’t help much without thinking through all parts

mikerod22:10:48

a brute force thing to me is to do lein-shell

mikerod22:10:58

although I know you have fears in travis

mikerod22:10:28

eh, shell sounds nasty actually

mikerod22:10:35

since you have to change the jar name, but also the pom inside of it

mikerod22:10:44

probably would have to come up with better plan than that

mikerod22:10:39

so I guess you’ve tried just changing the :version in the pom?

deactivateduser22:10:57

In the project.clj you mean, or post-process the generated pom.xml?

mikerod22:10:32

:profiles {:jdk-x-stuff {:version "1.0-jdk-x"
                         :java-cmd "jdk-x"}
           :jdk-y-stuff {:version "1.0-jdk-y"
                         :java-cmd "jdk-y"}}

mikerod22:10:47

there is a fairly good chance it doesn’t work

mikerod22:10:04

but when dealing with the project data in like a plugin, the version is just under the key :version

deactivateduser18:10:50

For anyone following along at home, here's where I eventually landed: https://github.com/fn-fx/fn-fx/blob/c3eb3eb07eaa4bdb0f18724f08df7596da69c8ce/project.clj #dirtyhacks

mikerod19:10:19

why would you have to add to the default profiles

mikerod19:10:27

you can just make an alias that puts with-profile on that

mikerod19:10:40

but the version-suffix thing doesn’t seem that unreasonable to me

mikerod19:10:03

except you’d have to run something to run lein different times with differnet jvms

mikerod19:10:29

jvm-cmd seems to be a way to avoid that, but I haven’t had time to mess around enough to demonstrate, so I guess not too useful hah

mikerod19:10:39

thanks for posting anyways interesting to see

mikerod21:10:26

specifically :java-cmd