Clojurians Log v2

Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: clojure.core.async.timers/timeout-daemon. Threads running in the image generator are no longer running at image runtime. Prevent threads from starting during image generation, or a started thread from being included in the image.
The culprit object has been instantiated by the 'himmelsstuermer.test_runner' class initializer with the following trace:
	at java.lang.Thread.(Thread.java:1267)
	at clojure.core.async.impl.timers$fn__13733$fn__13734.invoke(timers.clj:53)
	at clojure.lang.Delay.realize(Delay.java:44)
	at clojure.lang.Delay.deref(Delay.java:59)
	at clojure.core$deref.invokeStatic(core.clj:2337)
	at clojure.core$deref.invoke(core.clj:2323)
	at clojure.core.async.impl.timers$timeout.invokeStatic(timers.clj:58)
	at clojure.core.async.impl.timers$timeout.invokePrim(timers.clj)
	at clojure.core.async$timeout.invokeStatic(async.clj:114)
	at clojure.core.async$timeout.invokePrim(async.clj)
	at superv.async$simple_supervisor$pending__18063.invoke(async.cljc:77)
	at superv.async$simple_supervisor.invokeStatic(async.cljc:67)
	at superv.async$simple_supervisor.doInvoke(async.cljc:48)
	at clojure.lang.RestFn.invoke(RestFn.java:400)
	at superv.async$fn__18085.invokeStatic(async.cljc:115)
	at superv.async$fn__18085.invoke(async.cljc:110)
	at superv.async__init.load(Unknown Source)
	at superv.async__init.(Unknown Source)
	at java.lang.Class.forName0(Unknown Source)
	at java.lang.Class.forName(Class.java:578)
	at java.lang.Class.forName(Class.java:557)
	at clojure.lang.RT.classForName(RT.java:2229)
	at clojure.lang.RT.classForName(RT.java:2238)
	at clojure.lang.RT.loadClassForName(RT.java:2257)
	at clojure.lang.RT.load(RT.java:469)
	at clojure.lang.RT.load(RT.java:444)
	at clojure.core$load$fn__6931.invoke(core.clj:6189)
	at clojure.core$load.invokeStatic(core.clj:6188)
	at clojure.core$load.doInvoke(core.clj:6172)
	at clojure.lang.RestFn.invoke(RestFn.java:411)
	at clojure.core$load_one.invokeStatic(core.clj:5961)
	at clojure.core$load_one.invoke(core.clj:5956)
	at clojure.core$load_lib$fn__6873.invoke(core.clj:6003)
	at clojure.core$load_lib.invokeStatic(core.clj:6002)
	at clojure.core$load_lib.doInvoke(core.clj:5981)
	at clojure.lang.RestFn.applyTo(RestFn.java:145)
	at clojure.core$apply.invokeStatic(core.clj:669)
	at clojure.core$load_libs.invokeStatic(core.clj:6044)
	at clojure.core$load_libs.doInvoke(core.clj:6028)
	at clojure.lang.RestFn.applyTo(RestFn.java:140)
	at clojure.core$apply.invokeStatic(core.clj:669)
	at clojure.core$require.invokeStatic(core.clj:6066)
	at clojure.core$require.doInvoke(core.clj:6066)
	at clojure.lang.RestFn.invoke(RestFn.java:485)
	at datahike.tools$loading__6812__auto____13267.invoke(tools.cljc:1)
	at datahike.tools__init.load(Unknown Source)
	at datahike.tools__init.(Unknown Source)

.......and long, long more...

Like I see in code of superv.async there is a try/catch block to not fail in native image build process... but..

Looks like this macro is not working correct:

(defmacro native-image-build? []
     (try
       (and (Class/forName "org.graalvm.nativeimage.ImageInfo")
            #_(eval '(org.graalvm.nativeimage.ImageInfo/inImageBuildtimeCode)))
       (catch Exception _
         false)))

Hmm, I see. Interesting.

We did not run into this yet, because we only compiled datahike directly to native images ourselves, I think.

How do you build your jar file?

If this Clojure code is evaluated when you create the jar, e.g. through AOT, then you will run into this obstacle. But if the superv.async namespace is evaluated in your native build context, when loaded form the jar file, then it should be fine.

How do you build your jar?

I am doing lein install and then using jar from local .m2 repo. I tried different :aot settings but it doesn't help. I guess because anyway from entry point we achieve a point of requiring datahike.

Can you not AOT at all? Maybe the AOT problem is in datahike.

You could try to clone datahike locally and put it into your deps, if our jar is broken. Maybe that helps when you build it.

No I can't not AOT...it's not starting then

Can you clone datahike and put it in your deps with :local/root?

And then native build directly in this environment?

I can, but... not right now.

> And then native build directly in this environment? What to build? Library? Or app?

Basically you need to make sure that datahike's Clojure code is on your classpath and you directly native compile that. Then the macro will pick up the right environment. If you first compile a jar for datahike then it will fail.

As seems to be the case for the one on clojars.

Okay I will try, but right now I solving another native image problem. It's so annoying 😞

Why do you need native-image?

I need fast cold-run

I see

I definitely want datahike to work there, this is why I put the effort in to make it natively compilable. Lmk if you need help.

Yes, sure. I bet we will solve it. But there are so many caveeats

ok I found that confusing because native-image builds from bytecode, it doesn't do compilation of Clojure code. The missing thing for me was that one needs to include org.graalvm.sdk/graal-sdk {:mvn/version "24.1.1"} in deps so that org.graalvm.nativeimage.ImageInfocan be found at compilation time

Ah, I see. Feel free to open a PR if you think this will help others as well.

I don't know what it means to compile superv.async in the native image context. The way I compile my other stacks with native-image is to compile my Clojure code with clojure.tools.build.api/compile-clj to .class files, which are given to native-image on the classpath. Reading Datahike's bb.edn it looks like Datahike's native-image compilation process is the same, as that's what clj.native-image does (compile the Clojure code, then run native-image with the resulting .class files). That means that since native-image-build? is a macro it will return a compile time constant of false, but if I make it a function instead then it builds and my app works. There must be something I haven't noticed about the compilation process.

Yes, so if superv.async is loaded and macro expanded in your native image compilation context it should work.

Excuse me, I am not so experienced in clj-cli . What command I need to run to generate target/classes folder. I tried clj -X:deps prep but it doesn't work

ChatGPT says: To generate the target/classes folder in a Clojure project using clj-cli, you’ll need to ensure that you're using tools.deps with a build alias that compiles the code. The clj -X:deps prep command is generally for preparing dependencies, but to compile Clojure code to generate target/classes, you can try the following steps: 1. Add a build alias to your deps.edn file, like so:

clojure
   :aliases
   {:build {:deps {org.clojure/tools.build {:mvn/version "0.8.3"}}
            :ns-aliases {b build}
            :exec-fn b/build}}
   

clojure
   (ns build
     (:require [clojure.tools.build.api :as b]))

   (defn build
     [_]
     (b/compile-clj {:basis (b/create-basis)
                     :src-dirs ["src"]
                     :class-dir "target/classes"}))
   

bash
   clj -X:build
   

I lack a bit of context here.

Are you trying to build datahike?

That's ok and I know that but in source code i haven't found preconfigured command. That mislead me. And sometimes is better to ask a live person who know things, because GPT talking too much wrong things (especially about native images haha)

Are you trying to build datahike?How else you want me to test it with "in place" sources?

It fails with classes not found if I just put the clojure sources in my project

Anyway.. I found a way to get rid of AOT and it works as expected. Thank you. But it was important (in my only case I guess) to put dependencies as this:

[org.clojure/core.async "1.6.681"]
[io.replikativ/datahike "0.6.1592"
 :exclude [org.clojure/core.async]]

I see. I guess we should bump the dependency on core.async then as well. Not fully sure what went wrong here, but I am glad it works. Lmk any changes you think need to be made.

Hmm... in my case this problem was because datahike was as a part of my framework, that was AOT compiled and used in app that was about to complie to native-image, but I do not think that @octo221 have same build-chain... when I made framework backend-agnostic and not dependent from datahike , and added datahike-backend implementation and datahike dependency directly to application that was natively-compiled, the error with (simple-supervisor) was gone

hi @sasha_bogdanov_dev, I'm having the same problem compiling with native image, Error: Detected a started Thread in the image heap. Thread name: async-dispatch-2. caused by:

at java.lang.Thread.<init>(Thread.java:1262)
	at clojure.core.async.impl.timers$fn__694$fn__695.invoke(timers.clj:53)
	at clojure.lang.Delay.realize(Delay.java:44)
	at clojure.lang.Delay.deref(Delay.java:59)
	at clojure.core$deref.invokeStatic(core.clj:2337)
	at clojure.core$deref.invoke(core.clj:2323)
	at clojure.core.async.impl.timers$timeout.invokeStatic(timers.clj:58)
	at clojure.core.async.impl.timers$timeout.invokePrim(timers.clj)
	at clojure.core.async$timeout.invokeStatic(async.clj:114)
	at clojure.core.async$timeout.invokePrim(async.clj)
	at superv.async$simple_supervisor$pending__8473.invoke(async.cljc:77)
	at superv.async$simple_supervisor.invokeStatic(async.cljc:67)
	at superv.async$simple_supervisor.doInvoke(async.cljc:48)...
at datahike.tools$loading__6812__auto____228.invoke(tools.cljc:1)
	at datahike.tools__init.load(Unknown Source)
	at datahike.tools__init.<clinit>(Unknown Source)...
at datahike.config$loading__6812__auto____161.invoke(config.cljc:1)
	at datahike.config__init.load(Unknown Source)
	at datahike.config__init.<clinit>(Unknown Source)...
at datahike.db__init.load(Unknown Source)
	at datahike.db__init.<clinit>(Unknown Source)...

Something wrong in your code. Try to remove any code that can start a Thread from top-level of namespaces, because this code is launched on build-time

at the top level all it's doing is requiring datahike.api and datahike-dynamodb.core

So... i remember there was also a problem about dependencies conflict. I already moved to Java SnapStart runtime and can't find answer on your question immidiately. I will lurk down when I will have time, but I do not know when(((

oh I didn't know about SnapStart. Did you ever get it to compile with native image ?

I did but with many considerations

I get such errors if any code uses datahike.api

any code reachable from -main

Had you used --features=clj_easy.graal_build_time.InitClojureClasses native-image option?

yes yes

all my other NI stacks compile

maybe try this:

:dependencies [[org.clojure/core.async "1.6.681"]
                                        [io.replikativ/datahike "0.6.1592"
                                         :exclude [org.clojure/clojure
                                                   org.clojure/core.async]]]

(ns my.datahike.test
  (:gen-class)
  (:require
    [datahike.api :as d]))

(defn -main
    ([args]
     (println "-main" args)
     (let [c {:store {:backend :mem :id "default"}
              :name "db"}]
       (d/database-exists? c))
     (println "ok.")))

@octo221 You need to compile superv.async in the native image context, because it skips the thread then (the thread is providing a default supervisor that is also helpful while developing). Datahike compilation in bb.edn e.g. bb ni-cli compiles the native image in the right context. It could be that you have use local/root either for superv.async or datahike directly to ensure it is compiled in your context and not AOT. I think if you depend on datahike form maven it might do AOT for some reason.

It comes from https://github.com/replikativ/superv.async/blob/main/src/superv/async.cljc#L115