Fork me on GitHub
#depstar
<
2021-04-13
>
golanweiss07:04:06

Hi, I’m trying to build an uberjar for a clojure 1.8 project (yes, I know…) However the POM file that was automatically-created has a dependency of clojure 1.10.3 (which is the version of the clojure CLI I’m using)… Is this a known issue or am I doing something wrong?

clojure -X:uberjar :pom-file '"./my-app/pom.xml"' :sync-pom true :group-id xxx :artifact-id my-app :version '"1.0.2"' :aliases '[:defaults :my-app]' :repro false :jar my-app.jar :main-class my-app.main :aot true
And the pom file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
  <modelVersion>4.0.0</modelVersion>
  <packaging>jar</packaging>
  <groupId>xxx</groupId>
  <artifactId>my-app</artifactId>
  <version>1.0.2</version>
  <name>my-app</name>
  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.10.3</version>
    </dependency>
  </dependencies>
  <build>
    <sourceDirectory>src</sourceDirectory>
  </build>
  <repositories>
    <repository>
      <id>clojars</id>
      <url></url>
    </repository>
    <repository>
      <id>my-private-maven</id>
      <url></url>
    </repository>
  </repositories>
</project>

golanweiss09:04:08

Note the uberjar itself contains clojure 1.8 so it only looks like a “cosmetic” issue.

golanweiss15:04:33

It seems pom.xml is identical if I just run

clojure -Spom
But I thought I can add additional aliases. Even with
clojure -Spom -A:...
result is the same.

golanweiss15:04:15

so I think this is a bug with Spom I’m using a monorepo as described https://corfield.org/blog/2021/02/23/deps-edn-monorepo/ (thank you @seancorfield) and running my clojure CLI commands from a top root. Seems Spom assumes current working dir which in my case is… empty. Running Spom from within each nested module dir generates almost the correct pom.xml file, but still it is missing all my {:local/root ... deps.

Joshua Suskalo12:04:36

Is there a way to get depstar to not ignore .cljs files? I'd like to package this jar once and have it available both for JVM and JS Clojure variants.

Joshua Suskalo12:04:43

Oh, it seems like it's picking it up now

Joshua Suskalo12:04:54

I must've just not retried after I moved the cljs directory to the main classpath

golanweiss15:04:15
replied to a thread:Hi, I’m trying to build an uberjar for a clojure 1.8 project (yes, I know…) However the POM file that was automatically-created has a dependency of clojure 1.10.3 (which is the version of the clojure CLI I’m using)… Is this a known issue or am I doing something wrong? clojure -X:uberjar :pom-file '"./my-app/pom.xml"' :sync-pom true :group-id xxx :artifact-id my-app :version '"1.0.2"' :aliases '[:defaults :my-app]' :repro false :jar my-app.jar :main-class my-app.main :aot true And the pom file: &lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;project xmlns="<http://maven.apache.org/POM/4.0.0>" xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>" xsi:schemaLocation="<http://maven.apache.org/POM/4.0.0> <http://maven.apache.org/xsd/maven-4.0.0.xsd>"&gt; &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt; &lt;packaging&gt;jar&lt;/packaging&gt; &lt;groupId&gt;xxx&lt;/groupId&gt; &lt;artifactId&gt;my-app&lt;/artifactId&gt; &lt;version&gt;1.0.2&lt;/version&gt; &lt;name&gt;my-app&lt;/name&gt; &lt;dependencies&gt; &lt;dependency&gt; &lt;groupId&gt;org.clojure&lt;/groupId&gt; &lt;artifactId&gt;clojure&lt;/artifactId&gt; &lt;version&gt;1.10.3&lt;/version&gt; &lt;/dependency&gt; &lt;/dependencies&gt; &lt;build&gt; &lt;sourceDirectory&gt;src&lt;/sourceDirectory&gt; &lt;/build&gt; &lt;repositories&gt; &lt;repository&gt; &lt;id&gt;clojars&lt;/id&gt; &lt;url&gt;<https://repo.clojars.org/>&lt;/url&gt; &lt;/repository&gt; &lt;repository&gt; &lt;id&gt;my-private-maven&lt;/id&gt; &lt;url&gt;<s3://my-private-maven.com>&lt;/url&gt; &lt;/repository&gt; &lt;/repositories&gt; &lt;/project&gt;

so I think this is a bug with Spom I’m using a monorepo as described https://corfield.org/blog/2021/02/23/deps-edn-monorepo/ (thank you @seancorfield) and running my clojure CLI commands from a top root. Seems Spom assumes current working dir which in my case is… empty. Running Spom from within each nested module dir generates almost the correct pom.xml file, but still it is missing all my {:local/root ... deps.

seancorfield16:04:43

For clojure -Spom, you would need to include your aliases with -A before -Spom. I thought depstar used the same basis for calling t.d.a’s sync-pom as for building the JAR… can you create an issue on GH for depstar for me to take a look at @golan1w?

👍 2
markbastian16:04:28

I am trying to get the exclude parameter working correctly for my depstar target. As an example, I have the directories backend, scripts, and user at the root of my project. They are all being packaged into the uberjar and I don’t want that. I’ve tried the following to exclude them:

:exclude       ["scripts/*" "backend/*" "user/*"]
I’ve added these at both the alias definition level as well as under :exec-args without success. Any idea what I am doing wrong?

seancorfield17:04:41

Those aren’t valid regexes — they’re shell glob patterns.

seancorfield17:04:52

"scripts/.*" would be a regex.

seancorfield17:04:37

Well, I guess they are valid regexes but they’re not what you intended: "scripts/*" matches scripts followed by zero or more / characters.

seancorfield17:04:02

So scripts, scripts/, scripts//, scripts/// etc. @markbastian

seancorfield17:04:22

Happy to make the documentation clearer on that, if you have a suggestion?

markbastian17:04:29

Thanks! Yeah, I read it as regexes but was thrown off by the shell glob patterns as to if those worked or not. I’m still used to the shell patterns. As far as docs go, I think an example with directory exclusion could be good.

markbastian17:04:45

Is there a way to specify includes vs. excludes or set a target directory that will be uberjarred? I read all the options and didn’t see that. Having experience with maven uberjar or shade I was hoping to see a target or output directory that is the final thing that is jarred up.

seancorfield18:04:16

I’m not quite sure what you’re asking @markbastian You can specify a full path for the :jar option which is where the JAR goes, e.g., :jar '"/path/to/the.jar"'

seancorfield18:04:54

Not sure what you mean by “target directory”?

markbastian18:04:16

I mean setting the explicit target for what gets included in the jar. I was hoping to be able to only have what is in paths/extra-paths but I think that is just what is used for aot. What actually gets included in the jar appears to be everything that is not explicitly excluded besides the default excludes. It would be nice to set default includes or only use the specified paths.

seancorfield18:04:15

AOT is transitive.

seancorfield18:04:57

Are you building a JAR or an uberjar?

markbastian18:04:50

Building an uberjar, but does it make sense that every directory in the project is included in the uberjar, not just source directories? Just want to make sure I’m not doing something wrong.

golanweiss18:04:13

I’m moving from lein (`lein uberjar`) to dpes & depstar, and compared the resulting jars and they are identical as far as I can see. I see in the uberjar only the folders that were in the :path, in my case src and resources.

golanweiss19:04:27

@seancorfield in the README there is:

Run depstar with :sync-pom true :version '"x.y.z"' to update the <version> and SCM <tag> in the pom.xml file.
I couldn’t manage to do that. It seems only jar or uberjar are possible.
clojure -X:depstar :pom-file '"./app/pom.xml"' :sync-pom true :version '"1.1.1"'
No function found on command line or in :exec-fn

seancorfield19:04:38

Your :depstar alias does not have :exec-fn.

phronmophobic19:04:39

I ran into the same issue. You have to also create a jar when you want to update the pom

seancorfield19:04:03

;; build an uberjar (application) with AOT compilation by default:
  :uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.0.211"}}
            :exec-fn hf.depstar/uberjar
            :exec-args {:aot true}}
  ;; build a jar (library):
  :jar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.0.211"}}
        :exec-fn hf.depstar/jar
        :exec-args {}}
  ;; generic depstar alias, use with jar or uberjar function name:
  :depstar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.0.211"}}
            :ns-default hf.depstar
            :exec-args {}}
clojure -X:jar selects a function, clojure -X:uberjar selects a function, clojure -X:depstar requires you to specify the function yourself as either jar or uberjar

seancorfield19:04:40

That’s not a depstar issue — that’s a CLI usage issue.

phronmophobic19:04:07

Is there a way to update the pom without creating a jar? I've been using:

clojure -X:depstar jar :jar reveal-exception.jar :sync-pom true :version '"1.0.2"'

seancorfield19:04:47

No, depstar only updates the pom.xml if you are building a JAR or uberjar.

golanweiss19:04:19

ok that’s what I thought, I was just wondering because of this sentence in the README.

phronmophobic19:04:34

I also got stuck on the exact same step with the same error. Would you accept a pull request with an example usage?

seancorfield19:04:28

I’m not understanding what you think needs to be changed?

seancorfield19:04:45

It says “generic depstar alias, use with jar or uberjar function name”

seancorfield19:04:33

And then it gives examples for each:

clojure -X:uberjar :jar MyProject.jar
# or:
clojure -X:depstar uberjar :jar MyProject.jar
and
clojure -X:jar :jar MyLib.jar
# or:
clojure -X:depstar jar :jar MyLib.jar

seancorfield19:04:11

and then later on:

clojure -X:uberjar :jar MyProject.jar :aliases '[:webassets]'
# or:
clojure -X:depstar uberjar :jar MyProject.jar :aliases '[:webassets]'
and
clojure -X:uberjar :classpath "$(clojure -Spath -A:webassets)" :jar MyProject.jar
# or:
clojure -X:depstar uberjar :classpath "$(clojure -Spath -A:webassets)" :jar MyProject.jar

phronmophobic19:04:46

When I first was trying the clojure cli in place of lein, I was trying to follow the "My Deployment Process" section and got stuck on: > Run depstar with :sync-pom true :version '"x.y.z"' to update the <version> and SCM <tag> in the pom.xml file. I know that's a simple thing if you're familiar with the clojure cli, but I didn't know how to turn that sentence into a command at the time

seancorfield19:04:05

In the pom.xml section it has this example:

clojure -X:uberjar :sync-pom true \
        :group-id myname :artifact-id myproject \
        :version '"1.2.3"' :jar MyProject.jar

phronmophobic19:04:54

My suggestion is to add an example to that step in the "My Deployment Process section": Run depstar with :sync-pom true :version '"x.y.z"' to update the <version> and SCM <tag> in the pom.xml file.

clojure -X:depstar jar :jar MyProject.jar :sync-pom true :version '"1.2.3"'

seancorfield19:04:00

Oh, so just that one sentence in My Deployment Process is what is confusing you?

👍 2
golanweiss19:04:02

I know and have been using it. Because of this sentence, I thought there’s a way to set the version in the pom without building the JAR.

seancorfield19:04:05

OK, I’ll edit that.

seancorfield19:04:38

Updated.

🙏 2
toot 2
markbastian19:04:37

I’m wondering if there’s something wrong with my path configuration. I really do just want what’s in my :path entries from my project in the uberjar along with transitive dependencies.

golanweiss19:04:50

you can add clojure -Spath to analyze it

markbastian19:04:04

Good call, gonna try that.

👍 2
markbastian19:04:43

To be clear, is the “normal” behavior of the uberjar target to include only the contents of the :path entries or is it expected to include all contents of the project except for those specifically excluded?

seancorfield19:04:15

An uberjar will include everything that is needed to run as a standalone application.

seancorfield19:04:11

So, yes, it’s “normal” that an uberjar includes “everything”

seancorfield19:04:01

“every directory in the project is included in the uberjar, not just source directories” — whatever is on your :paths and :extra-paths per the aliases you provide, plus all dependencies from :deps and :extra-deps (again, per the aliases you provide) @markbastian

markbastian19:04:54

Hmm, i just added a non-pathed folder with some garbage data in a fresh deps project and uberjarred it and the folder didn’t look like it was included. I think my project that is including all the extra directories (which has a pretty hairy deps.edn file) has some configuration that is user error on my part. Gonna try simplifying it to a reduced case that works.

markbastian20:04:37

Ok, I have a reduced case that I can present. Using this deps.edn file I can successfully create a jar of about 5MB with the uberjar alias. This is a working deps.edn file for my case. It produces an uberjar with only src + deps with aot compilation.

{:paths     ["src/clj" "src/cljc"]

 :mvn/repos {"central"    {:url ""}
             "clojars"    {:url ""}
             "confluent"  {:url ""}
             "" {:url ""}}

 :deps      {com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.1"}
             org.clojure/data.json              {:mvn/version "2.0.2"}
             hashp/hashp                        {:mvn/version "0.2.0"}}

 :aliases   {:uberjar {:extra-deps {com.github.seancorfield/depstar {:mvn/version "2.0.211"}}
                       :exec-fn    hf.depstar/uberjar
                       :exec-args  {:jar        "small.jar"
                                    :main-class my.lambda.foo
                                    :aot        true}}}}

;ls -al shows small.jar to be ~5.26MB
I want to create a similar result in a larger, common deps.edn file as a subtask. My first try was to add the following to my main deps.edn file. It produces an AOT compilation failure. I would expect the replacement values to give me exactly the paths and deps from the working example.
:clj-lambda {:extra-deps    {com.github.seancorfield/depstar {:mvn/version "2.0.193"}}
             :exec-fn       hf.depstar/uberjar
             :replace-paths ["src/clj" "src/cljc"]
             :replace-deps  {hashp/hashp                        {:mvn/version "0.2.0"}
                             com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.1"}
                             org.clojure/data.json              {:mvn/version "2.0.2"}}
             :exec-args     {:jar        "big.jar"
                             :main-class my.lambda.foo
                             :aot        true}}
I then tried this. It does work, but produces a 150MB file which includes tons of content in my working directory that is not part of any :path field.
:lambda-backend
            {:extra-paths  []
             :replace-deps {hashp/hashp                        {:mvn/version "0.2.0"}
                            com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.1"}
                            org.clojure/data.json              {:mvn/version "2.0.2"}}}

:clj-lambda {:extra-deps    {com.github.seancorfield/depstar {:mvn/version "2.0.193"}}
             :exec-fn       hf.depstar/uberjar
             :replace-paths ["src/clj" "src/cljc"]
             :exec-args     {:jar        "big.jar"
                             :main-class my.lambda.foo
                             :aliases    [:lambda-backend]
                             :aot        true}}

;ls -al shows big.jar to be ~150MB
My suspicion is that perhaps I am not using :replace-* correctly, but I am not sure. Any ideas?

seancorfield20:04:31

Your suspicion is correct.

seancorfield20:04:22

:replace-deps (and :replace-paths) are used to set up an environment for a tool to run (like depstar) that is not “polluted” by your project’s environment. It’s why the depstar README says to use :replace-deps to specify depstar itself.

seancorfield20:04:04

When you run the tool, it will compute the project basis — reading the root deps.edn file, the user deps.edn file (only if you tell depstar :repro false), and the project deps.edn file. If you need depstar to use aliases for computing what should go into your JAR file, you pass :aliases [:your :alias :list] to depstar and it will use those aliases when computing the project basis.

seancorfield20:04:33

You pretty much never want :replace-* for anything except a tool that you are running.

seancorfield20:04:20

Think of it like this: * Running the tool requires a classpath that has the tool itself and very little else * At runtime, the tool itself will compute a different classpath that is used for whatever the tool is trying to do with your project (run tests, build a JAR, analyze dependencies, etc)

seancorfield20:04:13

That’s why depstar says to use

;; build an uberjar (application) with AOT compilation by default:
  :uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.0.211"}}
            :exec-fn hf.depstar/uberjar
            :exec-args {:aot true}}
for the tool — on its own.

seancorfield20:04:01

And then it talks about how depstar builds the classpath (for the JAR building) and says:

If you need to adjust the computed classpath, based on aliases, you can supply a vector of aliases to the :aliases exec argument of depstar. This classpath is used for both the AOT compilation process and for the JAR building.

For example, you can add web assets into an uberjar by including an alias in your project deps.edn:

{:paths ["src"]
 :aliases {:webassets {:extra-paths ["public-html"]}}}
Then invoke depstar with the chosen aliases:

clojure -X:uberjar :jar MyProject.jar :aliases '[:webassets]'
# or:
clojure -X:depstar uberjar :jar MyProject.jar :aliases '[:webassets]'

seancorfield20:04:50

(and this is how quite a few deps.edn-based tools work: they need a minimal classpath to run the tool itself and then they compute a more comprehensive classpath for the task they perform)

markbastian20:04:09

Ok, that makes a lot of sense. The alias entries are for the tool and the and the execution arguments are for the project/target that the tool is processing.

markbastian21:04:26

And with that knowledge I can now do this and it works:

:lambda-backend
            {:replace-paths ["src/clj" "src/cljc"]
             :replace-deps  {hashp/hashp                        {:mvn/version "0.2.0"}
                             com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.1"}
                             org.clojure/data.json              {:mvn/version "2.0.2"}}}

:clj-lambda {:extra-deps {com.github.seancorfield/depstar {:mvn/version "2.0.193"}}
             :exec-fn    hf.depstar/uberjar
             :exec-args  {:jar        "big.jar" ;Ends up only being about 5MB.
                          :main-class my.lambda.foo
                          :aliases    [:lambda-backend]
                          :aot        true}}
…and I know it seems very odd that I am replacing my project deps, but in this case that is exactly what I am trying to do. I have a known minimum set of dependencies that I need for slimmed down target for an aws lambda that won’t deploy when you hit a certain unzipped size. Thanks for the detailed explanation and help!

seancorfield21:04:15

You should still use :replace-deps for depstar itself.

👍 2
seancorfield21:04:13

But, yes, if you want an uberjar that contains “just” the `:lambda-backend and whatever ends up transitively compiled into that, you are doing it correctly.

🙏 2
richiardiandrea22:04:34

Hi there, I am trying to build an uberjar but I am running into the following:

$ clojure -X:uberjar :jar "~/tmp/target/setup.jar"
Unreadable arg: "~/tmp/target/setup.jar"
The ~/tmp/target dir exists and it is readable by the user that is running the command What problem can be?

seancorfield22:04:50

You are not quoting the EDN string correctly. Also, you cannot use the ~ expansion in a string.

seancorfield22:04:13

clojure -X:uberjar :jar '"/tmp/target/setup.jar"'

seancorfield22:04:57

From the README:

clojure -X:uberjar :jar '"/tmp/MyTempProject.jar"'

richiardiandrea22:04:21

uhm..subtle, trying

seancorfield22:04:22

That’s just how -X works — nothing specific to depstar

richiardiandrea22:04:51

indeed, thanks that worked