Been using the new 1.12 add-lib and friends, and noticed how using a clj-kondo/clj-kondo added with add-lib errors out if com.datomic/local is in deps as well.
Setup:
mkdir -p repro-add-libs-clj-kondo/src
cd repro-add-libs-clj-kondo
echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
echo '(ns user)' > src/user.clj
With add-lib :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> clj
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
Directly via deps :
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}'
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
(cc @borkdude)Probably via cheshire. You can try to add-lib just cheshire and repro that way to exclude the bigger clj-kondo as a cause
hmm, no, can't repro with just cheshire
I see transit in the stack trace
can you repro with my steps above? just wondering if it's something specific to my setup
yes I can repro with just clj-kondo
perhaps try isolating using transit
not sure what's going on, but if there's anything I can do in clj-kondo let me know
well I can just use deps directly, but I get the impression that these 2 usages should result in the same output so wondering if there's some wrong or some caveats with add-lib
I can take a look at it. when you add libs, the lib versions you already have in scope are fixed, and it's possible you might need some newer version for the new lib you've added. there are definitely jackson lib versions that have compatibility issues due to breaking changes, would not surprise me at all if that's what you're seeing here.
to look at it, I need your starting deps.edn
it's all there in the setup, self contained
sorry, didn't scroll back far enough :)
I actually can't repro the CNFE
interesting
what version of the CLI are you using? clj -version
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -version
Clojure CLI version 1.11.4.1474
I could repro it using:
$ clj --version
Clojure CLI version 1.11.4.1474I'm using same, didn't fail for me
did you run from the project dir?
(just double checking)
yes
I have stuff in ~/.clojure/deps.edn but just tried with -Srepro to ignore it and could still get the CNFE:
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
I'm on mac, in case it matters
I see a different set of deps loaded by add-lib (the return value)
in case it matters:
$ java --version
java 21.0.3 2024-04-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.3+7.1 (build 21.0.3+7-LTS-jvmci-23.1-b37, mixed mode, sharing)can you give me your clj -Srepro -Stree output
$ clj -Srepro -Stree
org.clojure/clojure 1.12.0-rc2
. org.clojure/spec.alpha 0.5.238
. org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
. com.datomic/client 1.0.138
. com.cognitect/anomalies 0.1.12
. com.datomic/client-api 1.0.69
. com.datomic/client-impl-shared 1.0.106
. com.cognitect/hmac-authn 0.1.211
. commons-codec/commons-codec 1.16.0
. com.cognitect/anomalies 0.1.12
. org.clojure/core.async 1.6.681
X com.cognitect/http-client 1.0.127 :excluded
. com.cognitect/transit-clj 1.0.333
. com.cognitect/transit-java 1.0.371
X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
. org.msgpack/msgpack 0.6.12
. com.googlecode.json-simple/json-simple 1.1.1
. org.javassist/javassist 3.18.1-GA
. javax.xml.bind/jaxb-api 2.4.0-b180830.0359
. javax.activation/javax.activation-api 1.2.0
. com.datomic/query-support 0.8.28
. org.clojure/core.async 1.6.681
. com.cognitect/anomalies 0.1.12
. com.datomic/query-support 0.8.28
. org.clojure/core.async 1.6.681
. org.clojure/tools.analyzer.jvm 1.2.3
. org.clojure/tools.analyzer.jvm 1.2.3
. org.clojure/tools.analyzer 1.1.1
. org.clojure/core.memoize 1.0.253
. org.ow2.asm/asm 9.2
. org.clojure/tools.reader 1.3.6
. org.clojure/tools.analyzer 1.1.1
. org.clojure/core.memoize 1.0.253
. org.clojure/core.cache 1.0.225
. org.clojure/core.cache 1.0.225
. org.clojure/data.priority-map 1.1.0
. org.clojure/data.priority-map 1.1.0
. org.ow2.asm/asm 9.2
. org.clojure/tools.reader 1.3.6
X org.clojure/spec.alpha 0.2.176 :older-version
X org.clojure/core.specs.alpha 0.2.44 :older-version
. com.datomic/client-impl-shared 1.0.106
. com.cognitect/anomalies 0.1.12
. org.clojure/core.async 1.6.681
. commons-io/commons-io 2.15.1
. com.datomic/query-stats 1.0.12
. org.fressian/fressian 0.6.8
. com.github.ben-manes.caffeine/caffeine 2.8.1
. org.checkerframework/checker-qual 3.1.0
. com.google.errorprone/error_prone_annotations 2.3.4
. org.checkerframework/checker-qual 3.1.0
. com.google.errorprone/error_prone_annotations 2.3.4
. com.datomic/io-stats 1.0.12
. com.datomic/client-api 1.0.69
. org.clojure/core.async 1.6.681org.clojure/clojure 1.12.0-rc2
. org.clojure/spec.alpha 0.5.238
. org.clojure/core.specs.alpha 0.4.74
com.datomic/local 1.0.285
. com.datomic/client 1.0.138
. com.cognitect/anomalies 0.1.12
. com.datomic/client-api 1.0.69
. com.datomic/client-impl-shared 1.0.106
. com.cognitect/hmac-authn 0.1.211
. commons-codec/commons-codec 1.16.0
. com.cognitect/anomalies 0.1.12
. org.clojure/core.async 1.6.681
X com.cognitect/http-client 1.0.127 :excluded
. com.cognitect/transit-clj 1.0.333
. com.cognitect/transit-java 1.0.371
X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded
. org.msgpack/msgpack 0.6.12
. com.googlecode.json-simple/json-simple 1.1.1
. org.javassist/javassist 3.18.1-GA
. javax.xml.bind/jaxb-api 2.4.0-b180830.0359
. javax.activation/javax.activation-api 1.2.0
. com.datomic/query-support 0.8.28
. org.clojure/core.async 1.6.681
. com.cognitect/anomalies 0.1.12
. com.datomic/query-support 0.8.28
. org.clojure/core.async 1.6.681
. org.clojure/tools.analyzer.jvm 1.2.3
. org.clojure/tools.analyzer.jvm 1.2.3
. org.clojure/tools.analyzer 1.1.1
. org.clojure/core.memoize 1.0.253
. org.ow2.asm/asm 9.2
. org.clojure/tools.reader 1.3.6
. org.clojure/tools.analyzer 1.1.1
. org.clojure/core.memoize 1.0.253
. org.clojure/core.cache 1.0.225
. org.clojure/core.cache 1.0.225
. org.clojure/data.priority-map 1.1.0
. org.clojure/data.priority-map 1.1.0
. org.ow2.asm/asm 9.2
. org.clojure/tools.reader 1.3.6
X org.clojure/spec.alpha 0.2.176 :older-version
X org.clojure/core.specs.alpha 0.2.44 :older-version
. com.datomic/client-impl-shared 1.0.106
. com.cognitect/anomalies 0.1.12
. org.clojure/core.async 1.6.681
. commons-io/commons-io 2.15.1
. com.datomic/query-stats 1.0.12
. org.fressian/fressian 0.6.8
. com.github.ben-manes.caffeine/caffeine 2.8.1
. org.checkerframework/checker-qual 3.1.0
. com.google.errorprone/error_prone_annotations 2.3.4
. org.checkerframework/checker-qual 3.1.0
. com.google.errorprone/error_prone_annotations 2.3.4
. com.datomic/io-stats 1.0.12
. com.datomic/client-api 1.0.69
. org.clojure/core.async 1.6.681
hah:
X com.fasterxml.jackson.core/jackson-core 2.14.2 :excluded> I see a different set of deps loaded by add-lib (the return value)
I see a different set depending on wether I use -Srepro but thats to be expected
is it? you have :deps in ~/.clojure/deps.edn ?
yes, I do:
:deps {;;
criterium/criterium {:mvn/version "0.4.6"}
;;
;; remember to do `(require 'hashp.core)`
hashp/hashp {:mvn/version "0.2.2"}
;;
vvvvalvalval/scope-capture {:mvn/version "0.3.3"}
;;
org.clojure/tools.trace {:mvn/version "0.8.0"}
;;
io.github.tonsky/clj-reload {:mvn/version "0.4.1"}
;;
datascript/datascript {:mvn/version "1.6.5"}
}
perhaps move them to a :dev alias or so. having deps in deps.edn can cause unexpected issues since those deps are always prioritized over whatever you have in your project
in the case of this repro I think -Srepro should ignore them, right?
yes
(but I agree that you should put them in an alias :)
I do have clojure itself in the the deps there, since I want to force myyself to use clojure 1.12 rc2 everywhere ;)
I cannot now see what I thought was different in my output, so I may just need more coffee
oh, I was comparing to your first output without -Srepro, but I match the later one
but I still don't get the CNFE
is there anything else that could be affecting the repro?
nothing coming to mind yet. stepping away for a meeting
I definitely have the class
I can repro inside a clojure docker container, for a clean env
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> docker run -it --rm clojure sh
# ll
sh: 1: ll: not found
# ls
clojure-2257487035549979519.edn hsperfdata_root target
# which clj
/usr/local/bin/clj
# mkdir -p repro-add-libs-clj-kondo/src
# cd repro-add-libs-clj-kondo
# echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} com.datomic/local {:mvn/version "1.0.285"}}}' > deps.edn
# echo '(ns user)' > src/user.clj
# clj
Downloading: com/datomic/local/1.0.285/local-1.0.285.pom from central
.... elided a lot of stuff ....
Downloading: org/clojure/core.async/1.6.681/core.async-1.6.681.jar from central
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=>
# ^C
# ^C
# exit
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]>when you get to this point, can you do com.fasterxml.jackson.core.JsonFactory to find the class, or not?
wondering if it is a timing thing with loading
huh that's interesting
[N] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (add-lib 'clj-kondo/clj-kondo {:mvn/version "2024.08.29"})
[babashka/fs borkdude/edamame borkdude/sci.impl.reflector cheshire/cheshire clj-kondo/clj-kondo com.cognitect/http-client com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.dataformat/jackson-dataformat-cbor com.fasterxml.jackson.dataformat/jackson-dataformat-smile com.github.javaparser/javaparser-core io.replikativ/datalog-parser nrepl/bencode org.babashka/sci org.babashka/sci.impl.types org.eclipse.jetty/jetty-client org.eclipse.jetty/jetty-http org.eclipse.jetty/jetty-io org.eclipse.jetty/jetty-util tigris/tigris]
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> com.fasterxml.jackson.core.JsonFactory
com.fasterxml.jackson.core.JsonFactory
user=> (type com.fasterxml.jackson.core.JsonFactory)
java.lang.Class
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>
look at how the error changes on the second time I call clj-kondo/run!
the class is there
can even call type on it, but clj-kondo still throws CNFE
what happens when you now do (require 'clj-kondo.core :reload-all) ? just a wild guess
something to do with kondo caching?
I just confirmed that sequence above: I got the same error Filipe got, but was able to type the classname, and then got a different error regarding that classname on retrying the clj-kondo/run! call per Filipe's REPL session.
(For me, that's on Ubuntu 20.04, WSL2, Windows)
And I still get that error after a reload:
user=> (require 'clj-kondo.core :reload-all)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=>can you also check what you get for this?
user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x131fcb6f "jar:file:/Users/alex.miller/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]that is, is the class in jackson-core jar?
user=> (.getResource (.getContextClassLoader (Thread/currentThread)) "com/fasterxml/jackson/core/JsonFactory.class")
#object[java.net.URL 0x18af5091 "jar:file:/home/sean/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.17.0/jackson-core-2.17.0.jar!/com/fasterxml/jackson/core/JsonFactory.class"]Is something in Kondo (or Transit) using its own class loader perhaps?
also, can someone drop the (pst *e) when you get the CNFE?
The first one or the second one? Or both?
both I guess (unless they're the same :)
transit is not doing anything special with classloaders as far as I recall
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
cognitect.transit/read (transit.clj:323)
cognitect.transit/read (transit.clj:319)
clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
clj-kondo.impl.cache/load-when-missing (cache.clj:115)
clj-kondo.impl.cache/load-when-missing (cache.clj:108)
clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
jdk.internal.loader.BuiltinClassLoader.loadClass (BuiltinClassLoader.java:641)
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:188)
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
Execution error (ClassNotFoundException) at com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl/createParser (ReaderFactory.java:150).
com.fasterxml.jackson.core.JsonFactory
user=> (pst *e)
RuntimeException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:154)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:110)
cognitect.transit/read (transit.clj:323)
cognitect.transit/read (transit.clj:319)
clj-kondo.impl.cache/from-cache-1/fn--4219 (cache.clj:32)
clj-kondo.impl.cache/from-cache-1 (cache.clj:31)
clj-kondo.impl.cache/from-cache-1 (cache.clj:20)
clj-kondo.impl.cache/load-when-missing (cache.clj:115)
clj-kondo.impl.cache/load-when-missing (cache.clj:108)
clj-kondo.impl.cache/sync-cache*/fn--4276/fn--4277/fn--4278 (cache.clj:161)
clojure.core.protocols/iterator-reduce! (protocols.clj:42)
Caused by:
NoClassDefFoundError com/fasterxml/jackson/core/JsonFactory
com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
Caused by:
ClassNotFoundException com.fasterxml.jackson.core.JsonFactory
com.cognitect.transit.impl.ReaderFactory$JsonReaderImpl.createParser (ReaderFactory.java:150)
com.cognitect.transit.impl.ReaderFactory$ReaderImpl.initialize (ReaderFactory.java:134)
nil
user=>
Yup, seeing the same here.
you might see difference from 1st to 2nd based on changing negative cache info in the classloaders themselves
but I suspect the real problem here is the classloader being used during the read from from the transit cache not reaching the dynamic classloader where the classes have been added
any hypothesis on why it doesn't repro on your setup?
no
exciting then π
did you try the docker repro?
I assume it will repro, but what does that buy me
if that one doesn't repro for you it's even more weird... then it's not about the contents of the system, but maybe about the physical characteristics
but I don't have any interesting conclusion from that either
I don't actually have docker installed on this computer, don't really want to do so atm
afaict I have no clj-kondo caches anywhere in my tree, is that also true for yall?
I don't see what this has to do with the above problem. clj-kondo always reads its bundled cache anyways (transit) so even if you don't have a cache on disk, stuff is being read
and in a clean docker repro it also occurs
same, no clj kondo cache
interesting data point: starting with clj-kondo and adding datomic-local via add-lib does not throw CNFE:
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master)> echo '{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.12.0-rc2"} clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}}' > deps.edn
[I] filipesilva@Filipes-MacBook-Pro ~/s/repro-add-libs-clj-kondo (master) [SIGINT]> clj -Srepro
Clojure 1.12.0-rc2
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (nil? (clj-kondo/run! {:lint ["src"] :config {:analysis true}}))
false
user=> (add-lib 'com.datomic/local {:mvn/version "1.0.285"})
[com.cognitect/anomalies com.cognitect/hmac-authn com.datomic/client com.datomic/client-api com.datomic/client-impl-shared com.datomic/io-stats com.datomic/local com.datomic/query-stats com.datomic/query-support com.github.ben-manes.caffeine/caffeine com.google.errorprone/error_prone_annotations commons-codec/commons-codec commons-io/commons-io org.checkerframework/checker-qual org.clojure/core.async org.clojure/core.cache org.clojure/core.memoize org.clojure/data.priority-map org.clojure/tools.analyzer org.clojure/tools.analyzer.jvm org.fressian/fressian]
user=> (require '[datomic.client.api :as d])
nil
user=> (def client (d/client {:server-type :datomic-local :storage-dir "/tmp/repro" :system "mem"}))
#'user/client
user=> (d/create-database client {:db-name "repro"})
true
Can you compare (:libs (clojure.java.basis/current-basis)) from those two paths
sorry what paths?
like, before and after adding clj-kondo in the original repro?
Yeah, basic at so you end up at the same place
that's starting with datomic, before adding clj-kondo, after adding clj-kondo
I guess better than libs you could check :classpath-roots
and that's starting with clj-kondo, adding datomic
Thatβs the only thing with cp ordering, not sure if that matters
ok gonna do it again but with classpath-roots
starting with datomic, add clj-kondo
start with clj-kondo, add datomic-local
starting with both datomic-local and clj-kondo on deps.edn
I was thinking about a way to take advantage of information at compile time to speed up PersistentHashMap creation, in cases where keys are all literals. I think it might be promising. Basically, if you have a map literal like
{:a (foo)
:b (foo)
:c (foo)
:d (foo)
:e (foo)}
(ignore the ArrayMap threshold of 8 keys just for briefness of example)
At compile time you can create a "template" that already has precomputed the shape of the resulting hashmap, and already knows which spots to put the values in. Perhaps it stores an int[][] , representing the path from root to where each value belongs. Something like... (highly simplified):
class PersistentHashMapTemplate {
int[][] paths;
PersistentHashMap original; // has all the same keys, but values are all null
public PersistentHashMapTemplate(Object[] keys) {
this.original = createMapWithNullValues(keys);
this.paths = doTheHardImplementationWorkOfFindingPaths();
}
PersistentHashMap create(Object[] vals) {
PersistentHashMap result = original.clone();
for (int i = 0; i < paths.length; i++) {
insertAtPath(result, paths[i], vals[i]);
}
return result;
}
}
Maybe that's not the final approach (using paths), one could imagine more optimized version, perhaps some specialized clone() method which, as it's walking through the tree cloning each node, it has on hand the valuess to inject into the clone. Or whatever.
So then the emitted code would be something like...
static final PersistentHashMapTemplate PERSISTENT_HASHMAP_TEMPLATE_0 =
new PersistentHashMapTemplate(new Symbol[]{Symbol.create("a"),Symbol.create("b"),Symbol.create("c"),Symbol.create("d"),Symbo.create("e")});
// at the site where the literal is:
PERSISTENT_HASHMAP_TEMPLATE_0.create(new Object[]{foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke(), foo.invoke()});
FWIW I was able to roughly double the performance again with the following approach: Create a static hashmap containing all the static key-vals, and in the case of static-key+dynamic-val, set the value to a sentinel object containing the index that the value will be at when called to create an instance of the map. This allows the hashmap instances to share as much structure as possible with the static "template"
Perhaps the more generic way to do this would be to make an optimized update-vals implementation which takes advantage of the HashMap structure, directly cloning each node and swapping out the value with (f value) rather (assoc! acc k (f v)) each element as it currently does.
Then you could I guess use a more simplified approach than the above. The "Original" could store the index of the value in the literal expression, and then when creating instances, swap out the index value for the value at the index. Something like:
(let [original {:a 0 :b 1 :c 2 :d 3 :e 4}]
values [(foo) (foo) (foo) (foo) (foo)]
(update-vals original (fn [i] (nth values i))))(actual implementation would be Java but you get the idea)
you should see my talk from JVMLS a few years ago, I talk about templates directly
Oh nice, link?
Very nice, thanks for that
Did a quick implementation, seems quite promising, like 400% speedup in the case of 10 keys
400% speedup... on 0.001% of runtime
Hehe yes. Not sure if there's an easy way to get an idea of how much time/memory is being spent intializing these maps in typical clojure programs
Pushed the prototype implementation up if people wanted to see https://github.com/joshlemer/clojure/pull/1/files?diff=split&w=1