clojure-dev

Filipe Silva 2024-08-30T13:39:46.446659Z

Been using the new 1.12 add-lib and friends, and noticed how using a clj-kondo/clj-kondo added with add-lib errors out if com.datomic/local is in deps as well. Setup:

mkdir -p repro-add-libs-clj-kondo/src
cd repro-add-libs-clj-kondo
echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
echo '(ns user)' > src/user.clj
With add-lib :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> clj
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
Directly via deps :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}'
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
(cc @borkdude)

borkdude 2024-08-30T13:41:25.287849Z

Probably via cheshire. You can try to add-lib just cheshire and repro that way to exclude the bigger clj-kondo as a cause

borkdude 2024-08-30T13:43:36.144069Z

hmm, no, can't repro with just cheshire

borkdude 2024-08-30T13:44:24.656649Z

I see transit in the stack trace

Filipe Silva 2024-08-30T13:45:44.692259Z

can you repro with my steps above? just wondering if it's something specific to my setup

borkdude 2024-08-30T13:56:15.520829Z

yes I can repro with just clj-kondo

borkdude 2024-08-30T13:56:23.541599Z

perhaps try isolating using transit

borkdude 2024-08-30T13:58:42.488289Z

not sure what's going on, but if there's anything I can do in clj-kondo let me know

Filipe Silva 2024-08-30T14:02:47.133769Z

well I can just use deps directly, but I get the impression that these 2 usages should result in the same output so wondering if there's some wrong or some caveats with add-lib

Alex Miller (Clojure team) 2024-08-30T14:07:27.404999Z

I can take a look at it. when you add libs, the lib versions you already have in scope are fixed, and it's possible you might need some newer version for the new lib you've added. there are definitely jackson lib versions that have compatibility issues due to breaking changes, would not surprise me at all if that's what you're seeing here.

Alex Miller (Clojure team) 2024-08-30T14:07:54.593539Z

to look at it, I need your starting deps.edn

Filipe Silva 2024-08-30T14:08:12.053899Z

it's all there in the setup, self contained

Alex Miller (Clojure team) 2024-08-30T14:08:25.966009Z

sorry, didn't scroll back far enough :)

Alex Miller (Clojure team) 2024-08-30T14:13:34.971279Z

I actually can't repro the CNFE

Filipe Silva 2024-08-30T14:13:52.813309Z

interesting

Alex Miller (Clojure team) 2024-08-30T14:14:46.050109Z

what version of the CLI are you using? clj -version

Filipe Silva 2024-08-30T14:15:03.671349Z

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -version
Clojure CLI version 1.11.4.1474

borkdude 2024-08-30T14:15:05.384079Z

I could repro it using:

$ clj --version
Clojure CLI version 1.11.4.1474

Alex Miller (Clojure team) 2024-08-30T14:15:41.691869Z

I'm using same, didn't fail for me

borkdude 2024-08-30T14:16:03.208299Z

did you run from the project dir?

borkdude 2024-08-30T14:16:09.503849Z

(just double checking)

Alex Miller (Clojure team) 2024-08-30T14:16:16.911069Z

yes

Filipe Silva 2024-08-30T14:16:54.544779Z

I have stuff in ~/.clojure/deps.edn but just tried with -Srepro to ignore it and could still get the CNFE:

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory

Filipe Silva 2024-08-30T14:17:28.775849Z

I'm on mac, in case it matters

Alex Miller (Clojure team) 2024-08-30T14:19:02.985999Z

I see a different set of deps loaded by add-lib (the return value)

borkdude 2024-08-30T14:19:24.272859Z

in case it matters:

$ java --version
java 21.0.3 2024-04-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37, mixed mode, sharing)

Alex Miller (Clojure team) 2024-08-30T14:19:49.150109Z

can you give me your clj -Srepro -Stree output

borkdude 2024-08-30T14:21:33.951299Z

$ clj -Srepro -Stree
org.clojure/clojure 1.12.0-rc2
  . org.clojure/spec.alpha 0.5.238
  . org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
  . com.datomic/client 1.0.138
    . com.cognitect/anomalies 0.1.12
    . com.datomic/client-api 1.0.69
    . com.datomic/client-impl-shared 1.0.106
      . com.cognitect/hmac-authn 0.1.211
        . commons-codec/commons-codec 1.16.0
        . com.cognitect/anomalies 0.1.12
        . org.clojure/core.async 1.6.681
      X com.cognitect/http-client 1.0.127 :excluded
      . com.cognitect/transit-clj 1.0.333
        . com.cognitect/transit-java 1.0.371
          X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
          . org.msgpack/msgpack 0.6.12
            . com.googlecode.json-simple/json-simple 1.1.1
            . org.javassist/javassist 3.18.1-GA
          . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
            . javax.activation/javax.activation-api 1.2.0
    . com.datomic/query-support 0.8.28
    . org.clojure/core.async 1.6.681
  . com.cognitect/anomalies 0.1.12
  . com.datomic/query-support 0.8.28
  . org.clojure/core.async 1.6.681
    . org.clojure/tools.analyzer.jvm 1.2.3
  . org.clojure/tools.analyzer.jvm 1.2.3
    . org.clojure/tools.analyzer 1.1.1
    . org.clojure/core.memoize 1.0.253
    . org.ow2.asm/asm 9.2
    . org.clojure/tools.reader 1.3.6
  . org.clojure/tools.analyzer 1.1.1
  . org.clojure/core.memoize 1.0.253
    . org.clojure/core.cache 1.0.225
  . org.clojure/core.cache 1.0.225
    . org.clojure/data.priority-map 1.1.0
  . org.clojure/data.priority-map 1.1.0
  . org.ow2.asm/asm 9.2
  . org.clojure/tools.reader 1.3.6
  X org.clojure/spec.alpha 0.2.176 :older-version
  X org.clojure/core.specs.alpha 0.2.44 :older-version
  . com.datomic/client-impl-shared 1.0.106
    . com.cognitect/anomalies 0.1.12
    . org.clojure/core.async 1.6.681
  . commons-io/commons-io 2.15.1
  . com.datomic/query-stats 1.0.12
  . org.fressian/fressian 0.6.8
  . com.github.ben-manes.caffeine/caffeine 2.8.1
    . org.checkerframework/checker-qual 3.1.0
    . com.google.errorprone/error_prone_annotations 2.3.4
  . org.checkerframework/checker-qual 3.1.0
  . com.google.errorprone/error_prone_annotations 2.3.4
  . com.datomic/io-stats 1.0.12
  . com.datomic/client-api 1.0.69
    . org.clojure/core.async 1.6.681

Filipe Silva 2024-08-30T14:22:06.785629Z

org.clojure/clojure 1.12.0-rc2
  . org.clojure/spec.alpha 0.5.238
  . org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
  . com.datomic/client 1.0.138
    . com.cognitect/anomalies 0.1.12
    . com.datomic/client-api 1.0.69
    . com.datomic/client-impl-shared 1.0.106
      . com.cognitect/hmac-authn 0.1.211
        . commons-codec/commons-codec 1.16.0
        . com.cognitect/anomalies 0.1.12
        . org.clojure/core.async 1.6.681
      X com.cognitect/http-client 1.0.127 :excluded
      . com.cognitect/transit-clj 1.0.333
        . com.cognitect/transit-java 1.0.371
          X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
          . org.msgpack/msgpack 0.6.12
            . com.googlecode.json-simple/json-simple 1.1.1
            . org.javassist/javassist 3.18.1-GA
          . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
            . javax.activation/javax.activation-api 1.2.0
    . com.datomic/query-support 0.8.28
    . org.clojure/core.async 1.6.681
  . com.cognitect/anomalies 0.1.12
  . com.datomic/query-support 0.8.28
  . org.clojure/core.async 1.6.681
    . org.clojure/tools.analyzer.jvm 1.2.3
  . org.clojure/tools.analyzer.jvm 1.2.3
    . org.clojure/tools.analyzer 1.1.1
    . org.clojure/core.memoize 1.0.253
    . org.ow2.asm/asm 9.2
    . org.clojure/tools.reader 1.3.6
  . org.clojure/tools.analyzer 1.1.1
  . org.clojure/core.memoize 1.0.253
    . org.clojure/core.cache 1.0.225
  . org.clojure/core.cache 1.0.225
    . org.clojure/data.priority-map 1.1.0
  . org.clojure/data.priority-map 1.1.0
  . org.ow2.asm/asm 9.2
  . org.clojure/tools.reader 1.3.6
  X org.clojure/spec.alpha 0.2.176 :older-version
  X org.clojure/core.specs.alpha 0.2.44 :older-version
  . com.datomic/client-impl-shared 1.0.106
    . com.cognitect/anomalies 0.1.12
    . org.clojure/core.async 1.6.681
  . commons-io/commons-io 2.15.1
  . com.datomic/query-stats 1.0.12
  . org.fressian/fressian 0.6.8
  . com.github.ben-manes.caffeine/caffeine 2.8.1
    . org.checkerframework/checker-qual 3.1.0
    . com.google.errorprone/error_prone_annotations 2.3.4
  . org.checkerframework/checker-qual 3.1.0
  . com.google.errorprone/error_prone_annotations 2.3.4
  . com.datomic/io-stats 1.0.12
  . com.datomic/client-api 1.0.69
    . org.clojure/core.async 1.6.681

borkdude 2024-08-30T14:22:06.898719Z

hah:

X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded

Filipe Silva 2024-08-30T14:22:58.047559Z

> I see a different set of deps loaded by add-lib (the return value) I see a different set depending on wether I use -Srepro but thats to be expected

Alex Miller (Clojure team) 2024-08-30T14:25:12.258839Z

is it? you have :deps in ~/.clojure/deps.edn ?

Filipe Silva 2024-08-30T14:26:01.642569Z

yes, I do:

:deps {;; 
        criterium/criterium         {:mvn/version "0.4.6"}
        ;; 
        ;; remember to do `(require 'hashp.core)`
        hashp/hashp                 {:mvn/version "0.2.2"}
        ;; 
        vvvvalvalval/scope-capture  {:mvn/version "0.3.3"}
        ;; 
        org.clojure/tools.trace     {:mvn/version "0.8.0"}
        ;; 
        io.github.tonsky/clj-reload {:mvn/version "0.4.1"}
        ;; 
        datascript/datascript {:mvn/version "1.6.5"}
        }

borkdude 2024-08-30T14:27:20.394089Z

perhaps move them to a :dev alias or so. having deps in deps.edn can cause unexpected issues since those deps are always prioritized over whatever you have in your project

πŸ‘ 1
Filipe Silva 2024-08-30T14:28:26.076829Z

in the case of this repro I think -Srepro should ignore them, right?

Alex Miller (Clojure team) 2024-08-30T14:28:32.526939Z

yes

Alex Miller (Clojure team) 2024-08-30T14:28:47.557619Z

(but I agree that you should put them in an alias :)

borkdude 2024-08-30T14:29:42.202139Z

I do have clojure itself in the the deps there, since I want to force myyself to use clojure 1.12 rc2 everywhere ;)

Alex Miller (Clojure team) 2024-08-30T14:29:57.429279Z

I cannot now see what I thought was different in my output, so I may just need more coffee

β˜• 1
Alex Miller (Clojure team) 2024-08-30T14:30:39.380659Z

oh, I was comparing to your first output without -Srepro, but I match the later one

Alex Miller (Clojure team) 2024-08-30T14:31:13.998969Z

but I still don't get the CNFE

Filipe Silva 2024-08-30T14:32:22.576599Z

is there anything else that could be affecting the repro?

Alex Miller (Clojure team) 2024-08-30T14:33:26.439109Z

nothing coming to mind yet. stepping away for a meeting

Alex Miller (Clojure team) 2024-08-30T14:33:55.152309Z

I definitely have the class

Filipe Silva 2024-08-30T14:40:18.468409Z

I can repro inside a clojure docker container, for a clean env

Filipe Silva 2024-08-30T14:40:41.139119Z

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> docker run -it --rm clojure sh


# ll
sh: 1: ll: not found
# ls
clojure-2257487035549979519.edn  hsperfdata_root  target
# which clj
/usr/local/bin/clj
# mkdir -p repro-add-libs-clj-kondo/src
# cd repro-add-libs-clj-kondo
# echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
# echo '(ns user)' > src/user.clj
# clj

Downloading: com/datomic/local/1.0.285/local-1.0.285.pom from central
 .... elided a lot of stuff ....
Downloading: org/clojure/core.async/1.6.681/core.async-1.6.681.jar from central
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=>
# ^C
# ^C
# exit
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]>

Alex Miller (Clojure team) 2024-08-30T18:14:10.491609Z

when you get to this point, can you do com.fasterxml.jackson.core.JsonFactory to find the class, or not?

Alex Miller (Clojure team) 2024-08-30T18:14:25.726299Z

wondering if it is a timing thing with loading

Filipe Silva 2024-08-30T18:41:32.254049Z

huh that's interesting

Filipe Silva 2024-08-30T18:41:34.143319Z

[N] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> com.fasterxml.jackson.core.JsonFactory
com.fasterxml.jackson.core.JsonFactory
user=> (type com.fasterxml.jackson.core.JsonFactory)
java.lang.Class
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>

Filipe Silva 2024-08-30T18:41:54.764589Z

look at how the error changes on the second time I call clj-kondo/run!

Filipe Silva 2024-08-30T18:42:03.922819Z

the class is there

Filipe Silva 2024-08-30T18:42:29.191829Z

can even call type on it, but clj-kondo still throws CNFE

borkdude 2024-08-30T18:43:11.707049Z

what happens when you now do (require 'clj-kondo.core :reload-all) ? just a wild guess

Alex Miller (Clojure team) 2024-08-30T18:43:43.303819Z

something to do with kondo caching?

❓ 1
seancorfield 2024-08-30T18:44:41.744199Z

I just confirmed that sequence above: I got the same error Filipe got, but was able to type the classname, and then got a different error regarding that classname on retrying the clj-kondo/run! call per Filipe's REPL session.

seancorfield 2024-08-30T18:45:03.930219Z

(For me, that's on Ubuntu 20.04, WSL2, Windows)

seancorfield 2024-08-30T18:45:49.310829Z

And I still get that error after a reload:

user=> (require 'clj-kondo.core :reload-all)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>

Alex Miller (Clojure team) 2024-08-30T18:52:29.508179Z

can you also check what you get for this?

user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x131fcb6f "jar:file:/Users/alex.miller/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]

Alex Miller (Clojure team) 2024-08-30T18:52:47.657579Z

that is, is the class in jackson-core jar?

seancorfield 2024-08-30T18:53:04.380669Z

user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x18af5091 "jar:file:/home/sean/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]

seancorfield 2024-08-30T18:53:43.536849Z

Is something in Kondo (or Transit) using its own class loader perhaps?

Alex Miller (Clojure team) 2024-08-30T18:53:50.658799Z

also, can someone drop the (pst *e) when you get the CNFE?

seancorfield 2024-08-30T18:54:07.079929Z

The first one or the second one? Or both?

Alex Miller (Clojure team) 2024-08-30T18:54:19.482889Z

both I guess (unless they're the same :)

Alex Miller (Clojure team) 2024-08-30T18:59:38.141399Z

transit is not doing anything special with classloaders as far as I recall

Filipe Silva 2024-08-30T19:05:59.148159Z

user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
	cognitect.transit/read (transit.clj:323)
	cognitect.transit/read (transit.clj:319)
	clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
	clj-kondo.impl.cache/load-when-missing (cache.clj:115)
	clj-kondo.impl.cache/load-when-missing (cache.clj:108)
	clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
	clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
	jdk.internal.loader.BuiltinClassLoader.loadClass (BuiltinClassLoader.java:641)
	jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:188)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
	cognitect.transit/read (transit.clj:323)
	cognitect.transit/read (transit.clj:319)
	clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
	clj-kondo.impl.cache/load-when-missing (cache.clj:115)
	clj-kondo.impl.cache/load-when-missing (cache.clj:108)
	clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
	clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
nil
user=>

seancorfield 2024-08-30T19:06:38.884919Z

Yup, seeing the same here.

Alex Miller (Clojure team) 2024-08-30T19:07:58.532799Z

you might see difference from 1st to 2nd based on changing negative cache info in the classloaders themselves

Alex Miller (Clojure team) 2024-08-30T19:09:06.605759Z

but I suspect the real problem here is the classloader being used during the read from from the transit cache not reaching the dynamic classloader where the classes have been added

Filipe Silva 2024-08-30T19:10:17.387549Z

any hypothesis on why it doesn't repro on your setup?

Alex Miller (Clojure team) 2024-08-30T19:10:22.167019Z

no

Filipe Silva 2024-08-30T19:10:39.285649Z

exciting then πŸ˜„

Filipe Silva 2024-08-30T19:11:23.936379Z

did you try the docker repro?

Alex Miller (Clojure team) 2024-08-30T19:12:19.260299Z

I assume it will repro, but what does that buy me

Filipe Silva 2024-08-30T19:12:44.036649Z

if that one doesn't repro for you it's even more weird... then it's not about the contents of the system, but maybe about the physical characteristics

Filipe Silva 2024-08-30T19:13:05.530539Z

but I don't have any interesting conclusion from that either

Alex Miller (Clojure team) 2024-08-30T19:14:23.839359Z

I don't actually have docker installed on this computer, don't really want to do so atm

πŸ‘ 1
Alex Miller (Clojure team) 2024-08-30T19:35:23.705339Z

afaict I have no clj-kondo caches anywhere in my tree, is that also true for yall?

borkdude 2024-08-30T19:41:40.784629Z

I don't see what this has to do with the above problem. clj-kondo always reads its bundled cache anyways (transit) so even if you don't have a cache on disk, stuff is being read

borkdude 2024-08-30T19:41:54.119789Z

and in a clean docker repro it also occurs

Filipe Silva 2024-08-30T19:52:48.080979Z

same, no clj kondo cache

Filipe Silva 2024-08-30T20:03:05.062869Z

interesting data point: starting with clj-kondo and adding datomic-local via add-lib does not throw CNFE:

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}' > deps.edn
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
user=> (add-lib 'com.datomic/local {:mvn/version "1.0.285"})
[com.cognitect/anomalies com.cognitect/hmac-authn com.datomic/client com.datomic/client-api com.datomic/client-impl-shared com.datomic/io-stats com.datomic/local com.datomic/query-stats com.datomic/query-support com.github.ben-manes.caffeine/caffeine com.google.errorprone/error_prone_annotations commons-codec/commons-codec commons-io/commons-io org.checkerframework/checker-qual org.clojure/core.async org.clojure/core.cache org.clojure/core.memoize org.clojure/data.priority-map org.clojure/tools.analyzer org.clojure/tools.analyzer.jvm org.fressian/fressian]
user=> (require '[datomic.client.api :as d])
nil
user=> (def client (d/client {:server-type :datomic-local :storage-dir "/tmp/repro" :system "mem"}))
#'user/client
user=> (d/create-database client {:db-name "repro"})
true

Alex Miller (Clojure team) 2024-08-30T20:05:20.073979Z

Can you compare (:libs (clojure.java.basis/current-basis)) from those two paths

Filipe Silva 2024-08-30T20:05:54.377729Z

sorry what paths?

Filipe Silva 2024-08-30T20:08:06.033309Z

like, before and after adding clj-kondo in the original repro?

Alex Miller (Clojure team) 2024-08-30T20:08:46.564549Z

Yeah, basic at so you end up at the same place

Filipe Silva 2024-08-30T20:09:21.137099Z

Filipe Silva 2024-08-30T20:09:48.349909Z

that's starting with datomic, before adding clj-kondo, after adding clj-kondo

Alex Miller (Clojure team) 2024-08-30T20:10:57.697249Z

I guess better than libs you could check :classpath-roots

Filipe Silva 2024-08-30T20:11:23.263289Z

Filipe Silva 2024-08-30T20:11:42.170039Z

and that's starting with clj-kondo, adding datomic

Alex Miller (Clojure team) 2024-08-30T20:11:52.681839Z

That’s the only thing with cp ordering, not sure if that matters

Filipe Silva 2024-08-30T20:12:29.559039Z

ok gonna do it again but with classpath-roots

Filipe Silva 2024-08-30T20:13:38.523989Z

starting with datomic, add clj-kondo

Filipe Silva 2024-08-30T20:14:37.811109Z

start with clj-kondo, add datomic-local

Filipe Silva 2024-08-30T20:15:55.351439Z

starting with both datomic-local and clj-kondo on deps.edn

JoshLemer 2024-08-30T17:23:52.521609Z

I was thinking about a way to take advantage of information at compile time to speed up PersistentHashMap creation, in cases where keys are all literals. I think it might be promising. Basically, if you have a map literal like

{:a (foo)
 :b (foo)
 :c (foo)
 :d (foo)
 :e (foo)}
(ignore the ArrayMap threshold of 8 keys just for briefness of example) At compile time you can create a "template" that already has precomputed the shape of the resulting hashmap, and already knows which spots to put the values in. Perhaps it stores an int[][] , representing the path from root to where each value belongs. Something like... (highly simplified):
class PersistentHashMapTemplate {
  int[][] paths;
  PersistentHashMap original; // has all the same keys, but values are all null

  public PersistentHashMapTemplate(Object[] keys) {
    this.original = createMapWithNullValues(keys);
    this.paths = doTheHardImplementationWorkOfFindingPaths();
  }

  PersistentHashMap create(Object[] vals) {
    PersistentHashMap result = original.clone();
    for (int i = 0; i < paths.length; i++) {
      insertAtPath(result, paths[i], vals[i]);
    }
    return result;
  }
}
Maybe that's not the final approach (using paths), one could imagine more optimized version, perhaps some specialized clone() method which, as it's walking through the tree cloning each node, it has on hand the valuess to inject into the clone. Or whatever. So then the emitted code would be something like...
static final PersistentHashMapTemplate PERSISTENT_HASHMAP_TEMPLATE_0 = 
  new PersistentHashMapTemplate(new Symbol[]{Symbol.create("a"),Symbol.create("b"),Symbol.create("c"),Symbol.create("d"),Symbo.create("e")});

// at the site where the literal is:

PERSISTENT_HASHMAP_TEMPLATE_0.create(new Object[]{foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke()});

JoshLemer 2024-09-09T21:24:17.504529Z

FWIW I was able to roughly double the performance again with the following approach: Create a static hashmap containing all the static key-vals, and in the case of static-key+dynamic-val, set the value to a sentinel object containing the index that the value will be at when called to create an instance of the map. This allows the hashmap instances to share as much structure as possible with the static "template"

JoshLemer 2024-08-30T17:38:38.061349Z

Perhaps the more generic way to do this would be to make an optimized update-vals implementation which takes advantage of the HashMap structure, directly cloning each node and swapping out the value with (f value) rather (assoc! acc k (f v)) each element as it currently does. Then you could I guess use a more simplified approach than the above. The "Original" could store the index of the value in the literal expression, and then when creating instances, swap out the index value for the value at the index. Something like:

(let [original {:a 0 :b 1 :c 2 :d 3 :e 4}]
      values [(foo) (foo) (foo) (foo) (foo)]
  (update-vals original (fn [i] (nth values i))))

JoshLemer 2024-08-30T17:39:29.019779Z

(actual implementation would be Java but you get the idea)

ghadi 2024-08-30T18:08:43.319179Z

you should see my talk from JVMLS a few years ago, I talk about templates directly

JoshLemer 2024-08-30T18:08:59.201729Z

Oh nice, link?

ghadi 2024-08-30T18:09:48.752269Z

https://youtu.be/UbqNrZMbxBg?t=2188

JoshLemer 2024-08-30T18:23:52.259939Z

Very nice, thanks for that

JoshLemer 2024-08-30T20:14:06.742819Z

Did a quick implementation, seems quite promising, like 400% speedup in the case of 10 keys

JoshLemer 2024-08-30T20:15:18.222349Z

ghadi 2024-08-30T20:58:59.765779Z

400% speedup... on 0.001% of runtime

JoshLemer 2024-08-30T21:00:32.476529Z

Hehe yes. Not sure if there's an easy way to get an idea of how much time/memory is being spent intializing these maps in typical clojure programs

JoshLemer 2024-08-30T21:24:18.392669Z

Pushed the prototype implementation up if people wanted to see https://github.com/joshlemer/clojure/pull/1/files?diff=split&amp;w=1