Fork me on GitHub
#clojure
<
2023-02-26
>
craftybones02:02:03

Was there a link to a State of Clojure survey for 2023?

Alex Miller (Clojure team)03:02:20

No but it’s coming next week

craftybones14:02:14

Thanks! Somehow, I always seem to be the one that asks this question a week before it comes out. Happened last year as well and the year before. Or deja vu is a thing

Carsten Behring14:02:13

I have a weird issue with some transitive dependencies not visible in `clj -X:deps :aliases '[:test]' and therefore missing in the classpath. My deps.edn has this alias:

:aliases
 {
  :test
  {
   :extra-paths ["test"]
   :extra-deps {
                scicloj/scicloj.ml.smile {:mvn/version "7.0.642"}
  }
and in the dependency tree scicloj.ml.smile appears, but without any transitive dependency:
scicloj/scicloj.ml.smile 7.0.642    // nothing below it
scicloj/metamorph 0.2.3
  X org.clojure/spec.alpha 0.2.194 :older-version
and I have indeed ClassNotFound exceptions from missing transitive dependencies. The library clear shows dependencies in Clojars: https://clojars.org/scicloj/scicloj.ml.smile and the downloaded pom.xml does have them as well

Carsten Behring14:02:11

I can reproduce with a deps.edn which only has the problematic dependency:

{:deps
 {scicloj/scicloj.ml.smile {:mvn/version "7.0.642"}}}
clj -Stree
-->
org.clojure/clojure 1.11.1
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62
scicloj/scicloj.ml.smile 7.0.642

Carsten Behring14:02:29

Interesting, going to a previous version shows the full tree. "7.0.632" works as expected.

Carsten Behring15:02:07

using the latest works as well:

{:deps
 {scicloj/scicloj.ml.smile {:git/url ""
                            :git/sha "4ca8fbcc9092508ec36e129ac715c84710ab7b52"}}}

Carsten Behring15:02:27

as far as I can see, there was no important change in the deps.edn file across the last versions: An hint how to debug this ?

Carsten Behring15:02:04

I compared the pom files on disk of the 2 releases and they are identical except an added description and licence block. (which I indeed added recently) Can it be that this confuses Clojure when it reads the poms to get the transitive dependencies ?

diff ~/.m2/repository/scicloj/scicloj.ml.smile/7.0.632/scicloj.ml.smile-7.0.632.pom ~/.m2/repository/scicloj/scicloj.ml.smile/7.0.642/scicloj.ml.smile-7.0.642.pom
2,8c2,9
< <project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
<   <modelVersion>4.0.0</modelVersion>
<   <packaging>jar</packaging>
<   <groupId>scicloj</groupId>
<   <artifactId>scicloj.ml.smile</artifactId>
<   <version>7.0.632</version>
<   <name>scicloj.ml.smile</name>
---
> <project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
>   <description>Smile  plugin for </description>
>   <licenses>
>     <license>
>       <name>Eclipse Public License</name>
>       <url></url>
>     </license>
>   </licenses>
59a61,68
>   <groupId>scicloj</groupId>
>   <artifactId>scicloj.ml.smile</artifactId>
>   <name>scicloj.ml.smile</name>
>   <version>7.0.642</version>
>   <scm>
>     <connection>scm:git:</connection>
>     <url></url>
>   </scm>

Carsten Behring15:02:06

The problematic pom neither has the line: "<packaging>jar</packaging>" and the block

<groupId>scicloj</groupId>
  <artifactId>scicloj.ml.smile</artifactId>
  <name>scicloj.ml.smile</name>
  <version>7.0.642</version>
is now at the end of the file, while in the working version, it is in the beginning. Is Clojure behaving differently due to this "structural changes" of the pom ? (-> not seeing the transitive dependencies ?)

Carsten Behring15:02:25

I think my issue started when I started to use this snippet and a "template pom":

(b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-pom "template/pom.xml"
                :scm {:connection "scm:git:"
                      :url ""}
                :src-dirs ["src"]})
template.pom.xml being
<project xmlns=""
        xmlns:xsi=""
        xsi:schemaLocation=" ">
<description>Smile  plugin for </description>
<licenses>
    <license>
      <name>Eclipse Public License</name>
      <url></url>
    </license>
</licenses>
</project>

Alex Miller (Clojure team)15:02:06

I doubt that has anything to do with it

Carsten Behring15:02:34

Is there anything "wrong" in this ? It seem to result in a "different" final pom.xml on which Clojure fails to "detect" teh transitive dependencies. The final pom.xml is this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
  <description>Smile  plugin for </description>
  <licenses>
    <license>
      <name>Eclipse Public License</name>
      <url></url>
    </license>
  </licenses>
  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.11.1</version>
    </dependency>
    <dependency>
      <groupId>generateme</groupId>
      <artifactId>fastmath</artifactId>
      <version>2.1.8</version>
    </dependency>
    <dependency>
      <groupId>scicloj</groupId>
      <artifactId>tablecloth</artifactId>
      <version>6.102</version>
    </dependency>
    <dependency>
      <groupId>org.bytedeco</groupId>
      <artifactId>openblas</artifactId>
      <version>0.3.10-1.5.4</version>
    </dependency>
    <dependency>
      <groupId>org.bytedeco</groupId>
      <artifactId>openblas-platform</artifactId>
      <version>0.3.10-1.5.4</version>
    </dependency>
    <dependency>
      <groupId>scicloj</groupId>
      <artifactId></artifactId>
      <version>0.6.2</version>
    </dependency>
    <dependency>
      <groupId>metosin</groupId>
      <artifactId>malli</artifactId>
      <version>0.9.2</version>
    </dependency>
    <dependency>
      <groupId>org.clojars.haifengl</groupId>
      <artifactId>smile</artifactId>
      <version>2.6.0</version>
    </dependency>
  </dependencies>
  <build>
    <sourceDirectory>src</sourceDirectory>
  </build>
  <repositories>
    <repository>
      <id>clojars</id>
      <url></url>
    </repository>
  </repositories>
  <groupId>scicloj</groupId>
  <artifactId>scicloj.ml.smile</artifactId>
  <name>scicloj.ml.smile</name>
  <version>7.0.642</version>
  <scm>
    <connection>scm:git:</connection>
    <url></url>
  </scm>
</project>

Alex Miller (Clojure team)15:02:44

but the downloaded pom in my local repo does have them

Carsten Behring15:02:10

I tried the same "version" as git dependency and it works as well;

{:deps
 {scicloj/scicloj.ml.smile {:git/url ""
                            :git/tag "v7.0.642"
                            :git/sha "5eabe19"}}}

Carsten Behring15:02:26

So must be "the pom" or similar maven related.

Alex Miller (Clojure team)15:02:01

ah, the pom in the jar has no deps

Alex Miller (Clojure team)15:02:28

oh, nvm - looking in the wrong place

Alex Miller (Clojure team)16:02:19

if that's true, it's in the library code tools.deps is using

Alex Miller (Clojure team)16:02:39

having an expected order seems wrong to me for xml but ... who knows

Alex Miller (Clojure team)16:02:32

you might also double-check that the project and xmlns stuff is all correct. if it has the wrong schema somehow then those won't look like the correct namespaced xml elements

Carsten Behring16:02:49

I found the reason ...

Carsten Behring16:02:10

The problematic pom misses:

<modelVersion>4.0.0</modelVersion>

Carsten Behring16:02:29

adding that makes it work.

Alex Miller (Clojure team)16:02:03

I guess that makes sense

Carsten Behring16:02:59

It seems indeed "mandatory" without default. https://maven.apache.org/guides/introduction/introduction-to-the-pom.html so my pom is wrong, therefore the release is unusable . -> will fix that

Carsten Behring16:02:10

Potentially clojure.tools.build.api/write-pom could try to validate the pom while writing it. This might prevent this or other hard to debug issues. Or any tooling Clojure uses to read the pom, should maybe fail while reading a pom.xml without the "<modelVersion>4.0.0</modelVersion>" instead of silently not giving dependencies. Maybe some issue could be opened somewhere ?

Alex Miller (Clojure team)17:02:48

tools.deps is using maven libs for that, so I don't think that's easy to add (maybe it should really be an issue on one of the maven-resolver libs)

Alex Miller (Clojure team)17:02:00

write-pom hard-codes the modelVersion tag, so will always produce it

Carsten Behring15:02:07

But I use write-pom with a template and don't have the modeVersion. Now I added it to the template. So this way of using write-pom https://github.com/scicloj/scicloj.ml.smile/blob/0bfc7b7d06cc26ed713089d6bee959f33018a6fb/build.clj#L34 assumes that src-pomcontains modelVersion without checking it. It's difficult to say if there is an issue somewhere. The documentation of write-pom says that

The pom deps, dirs, and repos are either synced from
the src-pom or generated from the basis
without specifying more what "sync" from the src-pom means.

Alex Miller (Clojure team)15:02:09

That could be more precise for sure

Carsten Behring16:02:44

Maybe the added docu is enough. I would still think that some code should be added in the "then" branch here: https://github.com/clojure/tools.build/blob/21cb08cfadb2a37038c7cb6b0115992bcd0528db/src/main/clojure/clojure/tools/build/tasks/write_pom.clj#L229 for "symmetry reasons". The "gen-pom" (= the else branch) guaranties to produce a "valid pom", while the "then" branch does not, it assumes a valid src-pom Overall write-pom only guaranties to produce a valid pom in some cases, which in my view is "not enough".

Alex Miller (Clojure team)16:02:38

I'm not going to do anything more on this for now

Alex Miller (Clojure team)16:02:02

when you provide a template, you take on this responsibility

Carsten Behring15:02:04

I compared the pom files on disk of the 2 releases and they are identical except an added description and licence block. (which I indeed added recently) Can it be that this confuses Clojure when it reads the poms to get the transitive dependencies ?

diff ~/.m2/repository/scicloj/scicloj.ml.smile/7.0.632/scicloj.ml.smile-7.0.632.pom ~/.m2/repository/scicloj/scicloj.ml.smile/7.0.642/scicloj.ml.smile-7.0.642.pom
2,8c2,9
< <project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
<   <modelVersion>4.0.0</modelVersion>
<   <packaging>jar</packaging>
<   <groupId>scicloj</groupId>
<   <artifactId>scicloj.ml.smile</artifactId>
<   <version>7.0.632</version>
<   <name>scicloj.ml.smile</name>
---
> <project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
>   <description>Smile  plugin for </description>
>   <licenses>
>     <license>
>       <name>Eclipse Public License</name>
>       <url></url>
>     </license>
>   </licenses>
59a61,68
>   <groupId>scicloj</groupId>
>   <artifactId>scicloj.ml.smile</artifactId>
>   <name>scicloj.ml.smile</name>
>   <version>7.0.642</version>
>   <scm>
>     <connection>scm:git:</connection>
>     <url></url>
>   </scm>

Noah Bogart22:02:05

I have a large number of macro invocations for a macro that’s a little complex. I’d like to cache the macro calls across runs of the code (using clojure -M) to speed up start up time. Is this possible?

Sam Ritchie00:02:01

You could generate the macroexpansion with a function, then call a memoized version of the function from within defmacro?

phill10:02:17

The macro could write to a file, or send an entry to memcached, as a side-effect after doing its expensive computation, or read an earlier-written file or memcached entry to skip the computation.

👍 2
Sam Ritchie12:02:42

Ah, sorry, I missed the crucial detail of “across runs”

👍 2
Noah Bogart14:02:09

I think what I'm looking for is graalvm haha but that's a big step. Thanks for the ideas