clojure-dev 2024-08-30

Been using the new 1.12 add-lib and friends, and noticed how using a clj-kondo/clj-kondo added with add-lib errors out if com.datomic/local is in deps as well. Setup:

mkdir -p repro-add-libs-clj-kondo/src
cd repro-add-libs-clj-kondo
echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
echo '(ns user)' > src/user.clj
With add-lib :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> clj
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
Directly via deps :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}'
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
(cc @borkdude)

Probably via cheshire. You can try to add-lib just cheshire and repro that way to exclude the bigger clj-kondo as a cause

hmm, no, can't repro with just cheshire

I see transit in the stack trace

can you repro with my steps above? just wondering if it's something specific to my setup

yes I can repro with just clj-kondo

perhaps try isolating using transit

not sure what's going on, but if there's anything I can do in clj-kondo let me know

well I can just use deps directly, but I get the impression that these 2 usages should result in the same output so wondering if there's some wrong or some caveats with add-lib

Alex Miller (Clojure team) 2024-08-30T14:07:27.404999Z

I can take a look at it. when you add libs, the lib versions you already have in scope are fixed, and it's possible you might need some newer version for the new lib you've added. there are definitely jackson lib versions that have compatibility issues due to breaking changes, would not surprise me at all if that's what you're seeing here.

Alex Miller (Clojure team) 2024-08-30T14:07:54.593539Z

to look at it, I need your starting deps.edn

it's all there in the setup, self contained

Alex Miller (Clojure team) 2024-08-30T14:08:25.966009Z

sorry, didn't scroll back far enough :)

Alex Miller (Clojure team) 2024-08-30T14:13:34.971279Z

I actually can't repro the CNFE

interesting

Alex Miller (Clojure team) 2024-08-30T14:14:46.050109Z

what version of the CLI are you using? clj -version

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -version
Clojure CLI version 1.11.4.1474

I could repro it using:

$ clj --version
Clojure CLI version 1.11.4.1474

Alex Miller (Clojure team) 2024-08-30T14:15:41.691869Z

I'm using same, didn't fail for me

did you run from the project dir?

(just double checking)

Alex Miller (Clojure team) 2024-08-30T14:16:16.911069Z

yes

I have stuff in ~/.clojure/deps.edn but just tried with -Srepro to ignore it and could still get the CNFE:

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory

I'm on mac, in case it matters

Alex Miller (Clojure team) 2024-08-30T14:19:02.985999Z

I see a different set of deps loaded by add-lib (the return value)

in case it matters:

$ java --version
java 21.0.3 2024-04-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37, mixed mode, sharing)

Alex Miller (Clojure team) 2024-08-30T14:19:49.150109Z

can you give me your clj -Srepro -Stree output

$ clj -Srepro -Stree
org.clojure/clojure 1.12.0-rc2
  . org.clojure/spec.alpha 0.5.238
  . org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
  . com.datomic/client 1.0.138
    . com.cognitect/anomalies 0.1.12
    . com.datomic/client-api 1.0.69
    . com.datomic/client-impl-shared 1.0.106
      . com.cognitect/hmac-authn 0.1.211
        . commons-codec/commons-codec 1.16.0
        . com.cognitect/anomalies 0.1.12
        . org.clojure/core.async 1.6.681
      X com.cognitect/http-client 1.0.127 :excluded
      . com.cognitect/transit-clj 1.0.333
        . com.cognitect/transit-java 1.0.371
          X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
          . org.msgpack/msgpack 0.6.12
            . com.googlecode.json-simple/json-simple 1.1.1
            . org.javassist/javassist 3.18.1-GA
          . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
            . javax.activation/javax.activation-api 1.2.0
    . com.datomic/query-support 0.8.28
    . org.clojure/core.async 1.6.681
  . com.cognitect/anomalies 0.1.12
  . com.datomic/query-support 0.8.28
  . org.clojure/core.async 1.6.681
    . org.clojure/tools.analyzer.jvm 1.2.3
  . org.clojure/tools.analyzer.jvm 1.2.3
    . org.clojure/tools.analyzer 1.1.1
    . org.clojure/core.memoize 1.0.253
    . org.ow2.asm/asm 9.2
    . org.clojure/tools.reader 1.3.6
  . org.clojure/tools.analyzer 1.1.1
  . org.clojure/core.memoize 1.0.253
    . org.clojure/core.cache 1.0.225
  . org.clojure/core.cache 1.0.225
    . org.clojure/data.priority-map 1.1.0
  . org.clojure/data.priority-map 1.1.0
  . org.ow2.asm/asm 9.2
  . org.clojure/tools.reader 1.3.6
  X org.clojure/spec.alpha 0.2.176 :older-version
  X org.clojure/core.specs.alpha 0.2.44 :older-version
  . com.datomic/client-impl-shared 1.0.106
    . com.cognitect/anomalies 0.1.12
    . org.clojure/core.async 1.6.681
  . commons-io/commons-io 2.15.1
  . com.datomic/query-stats 1.0.12
  . org.fressian/fressian 0.6.8
  . com.github.ben-manes.caffeine/caffeine 2.8.1
    . org.checkerframework/checker-qual 3.1.0
    . com.google.errorprone/error_prone_annotations 2.3.4
  . org.checkerframework/checker-qual 3.1.0
  . com.google.errorprone/error_prone_annotations 2.3.4
  . com.datomic/io-stats 1.0.12
  . com.datomic/client-api 1.0.69
    . org.clojure/core.async 1.6.681

org.clojure/clojure 1.12.0-rc2
  . org.clojure/spec.alpha 0.5.238
  . org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
  . com.datomic/client 1.0.138
    . com.cognitect/anomalies 0.1.12
    . com.datomic/client-api 1.0.69
    . com.datomic/client-impl-shared 1.0.106
      . com.cognitect/hmac-authn 0.1.211
        . commons-codec/commons-codec 1.16.0
        . com.cognitect/anomalies 0.1.12
        . org.clojure/core.async 1.6.681
      X com.cognitect/http-client 1.0.127 :excluded
      . com.cognitect/transit-clj 1.0.333
        . com.cognitect/transit-java 1.0.371
          X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
          . org.msgpack/msgpack 0.6.12
            . com.googlecode.json-simple/json-simple 1.1.1
            . org.javassist/javassist 3.18.1-GA
          . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
            . javax.activation/javax.activation-api 1.2.0
    . com.datomic/query-support 0.8.28
    . org.clojure/core.async 1.6.681
  . com.cognitect/anomalies 0.1.12
  . com.datomic/query-support 0.8.28
  . org.clojure/core.async 1.6.681
    . org.clojure/tools.analyzer.jvm 1.2.3
  . org.clojure/tools.analyzer.jvm 1.2.3
    . org.clojure/tools.analyzer 1.1.1
    . org.clojure/core.memoize 1.0.253
    . org.ow2.asm/asm 9.2
    . org.clojure/tools.reader 1.3.6
  . org.clojure/tools.analyzer 1.1.1
  . org.clojure/core.memoize 1.0.253
    . org.clojure/core.cache 1.0.225
  . org.clojure/core.cache 1.0.225
    . org.clojure/data.priority-map 1.1.0
  . org.clojure/data.priority-map 1.1.0
  . org.ow2.asm/asm 9.2
  . org.clojure/tools.reader 1.3.6
  X org.clojure/spec.alpha 0.2.176 :older-version
  X org.clojure/core.specs.alpha 0.2.44 :older-version
  . com.datomic/client-impl-shared 1.0.106
    . com.cognitect/anomalies 0.1.12
    . org.clojure/core.async 1.6.681
  . commons-io/commons-io 2.15.1
  . com.datomic/query-stats 1.0.12
  . org.fressian/fressian 0.6.8
  . com.github.ben-manes.caffeine/caffeine 2.8.1
    . org.checkerframework/checker-qual 3.1.0
    . com.google.errorprone/error_prone_annotations 2.3.4
  . org.checkerframework/checker-qual 3.1.0
  . com.google.errorprone/error_prone_annotations 2.3.4
  . com.datomic/io-stats 1.0.12
  . com.datomic/client-api 1.0.69
    . org.clojure/core.async 1.6.681

hah:

X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded

> I see a different set of deps loaded by add-lib (the return value) I see a different set depending on wether I use -Srepro but thats to be expected

Alex Miller (Clojure team) 2024-08-30T14:25:12.258839Z

is it? you have :deps in ~/.clojure/deps.edn ?

yes, I do:

:deps {;; 
        criterium/criterium         {:mvn/version "0.4.6"}
        ;; 
        ;; remember to do `(require 'hashp.core)`
        hashp/hashp                 {:mvn/version "0.2.2"}
        ;; 
        vvvvalvalval/scope-capture  {:mvn/version "0.3.3"}
        ;; 
        org.clojure/tools.trace     {:mvn/version "0.8.0"}
        ;; 
        io.github.tonsky/clj-reload {:mvn/version "0.4.1"}
        ;; 
        datascript/datascript {:mvn/version "1.6.5"}
        }

perhaps move them to a :dev alias or so. having deps in deps.edn can cause unexpected issues since those deps are always prioritized over whatever you have in your project

πŸ‘ 1

in the case of this repro I think -Srepro should ignore them, right?

Alex Miller (Clojure team) 2024-08-30T14:28:32.526939Z

yes

Alex Miller (Clojure team) 2024-08-30T14:28:47.557619Z

(but I agree that you should put them in an alias :)

I do have clojure itself in the the deps there, since I want to force myyself to use clojure 1.12 rc2 everywhere ;)

Alex Miller (Clojure team) 2024-08-30T14:29:57.429279Z

I cannot now see what I thought was different in my output, so I may just need more coffee

β˜• 1
Alex Miller (Clojure team) 2024-08-30T14:30:39.380659Z

oh, I was comparing to your first output without -Srepro, but I match the later one

Alex Miller (Clojure team) 2024-08-30T14:31:13.998969Z

but I still don't get the CNFE

is there anything else that could be affecting the repro?

Alex Miller (Clojure team) 2024-08-30T14:33:26.439109Z

nothing coming to mind yet. stepping away for a meeting

Alex Miller (Clojure team) 2024-08-30T14:33:55.152309Z

I definitely have the class

I can repro inside a clojure docker container, for a clean env

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> docker run -it --rm clojure sh


# ll
sh: 1: ll: not found
# ls
clojure-2257487035549979519.edn  hsperfdata_root  target
# which clj
/usr/local/bin/clj
# mkdir -p repro-add-libs-clj-kondo/src
# cd repro-add-libs-clj-kondo
# echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
# echo '(ns user)' > src/user.clj
# clj

Downloading: com/datomic/local/1.0.285/local-1.0.285.pom from central
 .... elided a lot of stuff ....
Downloading: org/clojure/core.async/1.6.681/core.async-1.6.681.jar from central
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=>
# ^C
# ^C
# exit
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]>

Alex Miller (Clojure team) 2024-08-30T18:14:10.491609Z

when you get to this point, can you do com.fasterxml.jackson.core.JsonFactory to find the class, or not?

Alex Miller (Clojure team) 2024-08-30T18:14:25.726299Z

wondering if it is a timing thing with loading

huh that's interesting

[N] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> com.fasterxml.jackson.core.JsonFactory
com.fasterxml.jackson.core.JsonFactory
user=> (type com.fasterxml.jackson.core.JsonFactory)
java.lang.Class
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>

look at how the error changes on the second time I call clj-kondo/run!

the class is there

can even call type on it, but clj-kondo still throws CNFE

what happens when you now do (require 'clj-kondo.core :reload-all) ? just a wild guess

Alex Miller (Clojure team) 2024-08-30T18:43:43.303819Z

something to do with kondo caching?

❓ 1

I just confirmed that sequence above: I got the same error Filipe got, but was able to type the classname, and then got a different error regarding that classname on retrying the clj-kondo/run! call per Filipe's REPL session.

(For me, that's on Ubuntu 20.04, WSL2, Windows)

And I still get that error after a reload:

user=> (require 'clj-kondo.core :reload-all)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>

Alex Miller (Clojure team) 2024-08-30T18:52:29.508179Z

can you also check what you get for this?

user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x131fcb6f "jar:file:/Users/alex.miller/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]

Alex Miller (Clojure team) 2024-08-30T18:52:47.657579Z

that is, is the class in jackson-core jar?

user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x18af5091 "jar:file:/home/sean/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]

Is something in Kondo (or Transit) using its own class loader perhaps?

Alex Miller (Clojure team) 2024-08-30T18:53:50.658799Z

also, can someone drop the (pst *e) when you get the CNFE?

The first one or the second one? Or both?

Alex Miller (Clojure team) 2024-08-30T18:54:19.482889Z

both I guess (unless they're the same :)

Alex Miller (Clojure team) 2024-08-30T18:59:38.141399Z

transit is not doing anything special with classloaders as far as I recall

user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
	cognitect.transit/read (transit.clj:323)
	cognitect.transit/read (transit.clj:319)
	clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
	clj-kondo.impl.cache/load-when-missing (cache.clj:115)
	clj-kondo.impl.cache/load-when-missing (cache.clj:108)
	clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
	clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
	jdk.internal.loader.BuiltinClassLoader.loadClass (BuiltinClassLoader.java:641)
	jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:188)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
	cognitect.transit/read (transit.clj:323)
	cognitect.transit/read (transit.clj:319)
	clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
	clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
	clj-kondo.impl.cache/load-when-missing (cache.clj:115)
	clj-kondo.impl.cache/load-when-missing (cache.clj:108)
	clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
	clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
	com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
nil
user=>

Yup, seeing the same here.

Alex Miller (Clojure team) 2024-08-30T19:07:58.532799Z

you might see difference from 1st to 2nd based on changing negative cache info in the classloaders themselves

Alex Miller (Clojure team) 2024-08-30T19:09:06.605759Z

but I suspect the real problem here is the classloader being used during the read from from the transit cache not reaching the dynamic classloader where the classes have been added

any hypothesis on why it doesn't repro on your setup?

Alex Miller (Clojure team) 2024-08-30T19:10:22.167019Z

no

exciting then πŸ˜„

did you try the docker repro?

Alex Miller (Clojure team) 2024-08-30T19:12:19.260299Z

I assume it will repro, but what does that buy me

if that one doesn't repro for you it's even more weird... then it's not about the contents of the system, but maybe about the physical characteristics

but I don't have any interesting conclusion from that either

Alex Miller (Clojure team) 2024-08-30T19:14:23.839359Z

I don't actually have docker installed on this computer, don't really want to do so atm

πŸ‘ 1
Alex Miller (Clojure team) 2024-08-30T19:35:23.705339Z

afaict I have no clj-kondo caches anywhere in my tree, is that also true for yall?

I don't see what this has to do with the above problem. clj-kondo always reads its bundled cache anyways (transit) so even if you don't have a cache on disk, stuff is being read

and in a clean docker repro it also occurs

same, no clj kondo cache

interesting data point: starting with clj-kondo and adding datomic-local via add-lib does not throw CNFE:

[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}' > deps.edn
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
user=> (add-lib 'com.datomic/local {:mvn/version "1.0.285"})
[com.cognitect/anomalies com.cognitect/hmac-authn com.datomic/client com.datomic/client-api com.datomic/client-impl-shared com.datomic/io-stats com.datomic/local com.datomic/query-stats com.datomic/query-support com.github.ben-manes.caffeine/caffeine com.google.errorprone/error_prone_annotations commons-codec/commons-codec commons-io/commons-io org.checkerframework/checker-qual org.clojure/core.async org.clojure/core.cache org.clojure/core.memoize org.clojure/data.priority-map org.clojure/tools.analyzer org.clojure/tools.analyzer.jvm org.fressian/fressian]
user=> (require '[datomic.client.api :as d])
nil
user=> (def client (d/client {:server-type :datomic-local :storage-dir "/tmp/repro" :system "mem"}))
#'user/client
user=> (d/create-database client {:db-name "repro"})
true

Alex Miller (Clojure team) 2024-08-30T20:05:20.073979Z

Can you compare (:libs (clojure.java.basis/current-basis)) from those two paths

sorry what paths?

like, before and after adding clj-kondo in the original repro?

Alex Miller (Clojure team) 2024-08-30T20:08:46.564549Z

Yeah, basic at so you end up at the same place

that's starting with datomic, before adding clj-kondo, after adding clj-kondo

Alex Miller (Clojure team) 2024-08-30T20:10:57.697249Z

I guess better than libs you could check :classpath-roots

and that's starting with clj-kondo, adding datomic

Alex Miller (Clojure team) 2024-08-30T20:11:52.681839Z

That’s the only thing with cp ordering, not sure if that matters

ok gonna do it again but with classpath-roots

starting with datomic, add clj-kondo

start with clj-kondo, add datomic-local

starting with both datomic-local and clj-kondo on deps.edn

I was thinking about a way to take advantage of information at compile time to speed up PersistentHashMap creation, in cases where keys are all literals. I think it might be promising. Basically, if you have a map literal like

{:a (foo)
 :b (foo)
 :c (foo)
 :d (foo)
 :e (foo)}
(ignore the ArrayMap threshold of 8 keys just for briefness of example) At compile time you can create a "template" that already has precomputed the shape of the resulting hashmap, and already knows which spots to put the values in. Perhaps it stores an int[][] , representing the path from root to where each value belongs. Something like... (highly simplified):
class PersistentHashMapTemplate {
  int[][] paths;
  PersistentHashMap original; // has all the same keys, but values are all null

  public PersistentHashMapTemplate(Object[] keys) {
    this.original = createMapWithNullValues(keys);
    this.paths = doTheHardImplementationWorkOfFindingPaths();
  }

  PersistentHashMap create(Object[] vals) {
    PersistentHashMap result = original.clone();
    for (int i = 0; i < paths.length; i++) {
      insertAtPath(result, paths[i], vals[i]);
    }
    return result;
  }
}
Maybe that's not the final approach (using paths), one could imagine more optimized version, perhaps some specialized clone() method which, as it's walking through the tree cloning each node, it has on hand the valuess to inject into the clone. Or whatever. So then the emitted code would be something like...
static final PersistentHashMapTemplate PERSISTENT_HASHMAP_TEMPLATE_0 = 
  new PersistentHashMapTemplate(new Symbol[]{Symbol.create("a"),Symbol.create("b"),Symbol.create("c"),Symbol.create("d"),Symbo.create("e")});

// at the site where the literal is:

PERSISTENT_HASHMAP_TEMPLATE_0.create(new Object[]{foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke()});

FWIW I was able to roughly double the performance again with the following approach: Create a static hashmap containing all the static key-vals, and in the case of static-key+dynamic-val, set the value to a sentinel object containing the index that the value will be at when called to create an instance of the map. This allows the hashmap instances to share as much structure as possible with the static "template"

Perhaps the more generic way to do this would be to make an optimized update-vals implementation which takes advantage of the HashMap structure, directly cloning each node and swapping out the value with (f value) rather (assoc! acc k (f v)) each element as it currently does. Then you could I guess use a more simplified approach than the above. The "Original" could store the index of the value in the literal expression, and then when creating instances, swap out the index value for the value at the index. Something like:

(let [original {:a 0 :b 1 :c 2 :d 3 :e 4}]
      values [(foo) (foo) (foo) (foo) (foo)]
  (update-vals original (fn [i] (nth values i))))

(actual implementation would be Java but you get the idea)

you should see my talk from JVMLS a few years ago, I talk about templates directly

Oh nice, link?

Very nice, thanks for that

Did a quick implementation, seems quite promising, like 400% speedup in the case of 10 keys

400% speedup... on 0.001% of runtime

Hehe yes. Not sure if there's an easy way to get an idea of how much time/memory is being spent intializing these maps in typical clojure programs

Pushed the prototype implementation up if people wanted to see https://github.com/joshlemer/clojure/pull/1/files?diff=split&amp;w=1