graalvm

simongray 2022-05-19T08:19:38.891659Z

Does anyone here know why VisualVM also exists bundled with GraalVM? The website doesnโ€™t really explain the split. https://visualvm.github.io/

borkdude 2022-05-19T08:21:34.378679Z

Perhaps just for convenience?

borkdude 2022-05-19T08:21:57.206499Z

Or perhaps their architecture requires specific changes to it? Don't know. You could ask on their dedicated slack

simongray 2022-05-19T08:22:33.548649Z

yeah, maybe I will do that.

littleli 2022-05-19T15:10:57.720989Z

My suggestion is that split is required because of different JIT engines in charge between openjdk and graalvm which may require additional treatment from the observability point of view.

littleli 2022-05-19T15:11:46.599129Z

C1, C2 vs Graal (?)

grounded_sage 2022-05-19T03:02:53.280729Z

Hey everyone. I'm curious what is involved to get Datahike compiling on the main branch. Does anyone know how I would pick apart this error?

Fatal error: org.graalvm.compiler.debug.GraalError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image runtime.  To see how this object got instantiated use --trace-object-instantiation=java.lang.Thread. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
	at com.oracle.graal.pointsto.util.AnalysisFuture.setException(AnalysisFuture.java:49)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:269)
	at com.oracle.graal.pointsto.util.AnalysisFuture.ensureDone(AnalysisFuture.java:63)
	at com.oracle.graal.pointsto.heap.ImageHeapScanner.lambda$postTask$9(ImageHeapScanner.java:611)
	at com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:193)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:177)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image runtime.  To see how this object got instantiated use --trace-object-instantiation=java.lang.Thread. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
	at com.oracle.svm.hosted.image.DisallowedImageHeapObjectFeature.error(DisallowedImageHeapObjectFeature.java:173)
	at com.oracle.svm.core.image.DisallowedImageHeapObjects.check(DisallowedImageHeapObjects.java:74)
	at com.oracle.svm.hosted.image.DisallowedImageHeapObjectFeature.replacer(DisallowedImageHeapObjectFeature.java:149)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.replaceObject(AnalysisUniverse.java:582)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.replaceObject(AnalysisConstantReflectionProvider.java:257)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.interceptValue(AnalysisConstantReflectionProvider.java:228)
	at com.oracle.svm.hosted.heap.SVMImageHeapScanner.transformFieldValue(SVMImageHeapScanner.java:126)
	at com.oracle.graal.pointsto.heap.ImageHeapScanner.onFieldValueReachable(ImageHeapScanner.java:331)
	at com.oracle.graal.pointsto.heap.ImageHeapScanner.lambda$createImageHeapObject$3(ImageHeapScanner.java:272)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 10 more

borkdude 2022-05-19T08:21:05.452569Z

This means that a thread is started on the top level. E.g.:

(def x (future :hello))
triggers such an error. Have you tried following up on the suggestion in the error message?

grounded_sage 2022-05-19T08:25:30.506619Z

yes I tried doing --initialize-at-run-time=... but think I am not selecting the classes properly. When I use the library https://clojars.org/com.github.clj-easy/graal-build-time I get a similar recommendation to initialize the class at runtime.

Error: Classes that should be initialized at run time got initialized during image building:
org.eclipse.jetty.util.Jetty was unintentionally initialized at build time. To see why org.eclipse.jetty.util.Jetty got initialized use --trace-class-initialization=org.eclipse.jetty.util.Jetty

borkdude 2022-05-19T08:26:34.279839Z

You should add --initialize-at-build-time=org.eclipse.jetty if you are instantiating jetty on the top level

borkdude 2022-05-19T08:26:42.188749Z

or push that to a delay or function

borkdude 2022-05-19T08:26:54.542179Z

why do you need jetty btw?

grounded_sage 2022-05-19T08:27:26.910699Z

running a server when starting the native executable

borkdude 2022-05-19T08:28:12.802199Z

tell me what this has to do with datahike? try to use the least amount of dependencies possible when getting started with a native image

borkdude 2022-05-19T08:28:40.505849Z

if you want a server that works out of the box with graalvm native image, try org.httpkit.server

grounded_sage 2022-05-19T08:31:05.286619Z

I thought that jetty was supported with graalvm native image. https://github.com/clj-easy/graalvm-clojure/tree/master/ring-jetty

borkdude 2022-05-19T08:32:06.905899Z

ok, perhaps, but you need to include the extra build time arguments then. https://app.slack.com/client/T03RZGPFR/threads?cdn_fallback=1#:~:text=jetty.util.Jetty-,5m,-borkdude

borkdude 2022-05-19T08:33:39.171309Z

the example in the graal-clojure repo uses --initialize-at-build-time which is now deprecated, you need to include the full list of package prefixes

borkdude 2022-05-19T08:33:59.005329Z

but you can cheat with --initialize-at-build-time=. - don't tell anyone ;)

๐Ÿ˜‚ 1
grounded_sage 2022-05-19T08:34:01.147819Z

yea I am trying to get datahike compiling using native image the server I am also interested in / partly also using the server to easily test it.

grounded_sage 2022-05-19T08:35:19.501929Z

the graal-build-time makes it default to that again right? so you would then only have to specify the runtime ones?

borkdude 2022-05-19T08:35:51.785979Z

no, graal-build-time only specifies clojure packages, you still have to specify all Java packages yourself

borkdude 2022-05-19T08:36:19.196499Z

in general you should try to keep the list as small as possible

grounded_sage 2022-05-19T08:36:21.370089Z

oh! and jetty is not

borkdude 2022-05-19T08:36:26.081359Z

yeah

grounded_sage 2022-05-19T08:40:16.908779Z

so I am basically using Jetty because that is the recommended starter by Eric Normand and http-kit was in the give it a pass. https://ericnormand.me/mini-guide/clojure-web-servers

borkdude 2022-05-19T08:40:52.741149Z

Don't just believe everything people say :)

borkdude 2022-05-19T08:41:07.533179Z

It depends on your use case

borkdude 2022-05-19T08:41:36.470619Z

But you can continue with jetty if you want of course, just add those arguments

grounded_sage 2022-05-19T08:41:43.799529Z

haha. Yea I have had trouble finding proper comparisons of the different servers.

grounded_sage 2022-05-19T08:50:08.719929Z

so this --initialize-at-build-time=org.eclipse.jetty works though I am not sure how to get this to work for superv.async

borkdude 2022-05-19T08:51:25.340639Z

what is superv.async

grounded_sage 2022-05-19T08:52:15.896179Z

the first one is just the org from the ring-jetty-adapter . Doing that with io.replikativ doesn't make much sense because there is also konserve etc. It says to do it with the class name. But when I look in the target directory classes I can't see how the jetty one maps to anything there.

grounded_sage 2022-05-19T08:52:54.237949Z

It's a library that provides Erlang style error handling. https://github.com/replikativ/superv.async

borkdude 2022-05-19T08:52:57.028359Z

Doesn't that lib already have support for graalvm? https://github.com/replikativ/superv.async/blob/5c1ac415b9642e5fb53dccca939dd1a9d44b76ad/src/superv/async.cljc#L113

grounded_sage 2022-05-19T08:53:21.295059Z

I'm still getting that build error for some reason.

borkdude 2022-05-19T08:53:36.759139Z

are you using the correct version?

borkdude 2022-05-19T08:53:54.145769Z

and what error exactly?

grounded_sage 2022-05-19T08:54:19.992569Z

yea I have been working up and down the stack. Even just trying to run superv.async

grounded_sage 2022-05-19T08:54:44.923419Z

the one of top level of thread here. The thread error

borkdude 2022-05-19T08:55:10.394759Z

That indicates that a thread is being started from the top level. Maybe the native image check isn't working anymore

grounded_sage 2022-05-19T08:55:54.633529Z

I'm not using it actually.

grounded_sage 2022-05-19T08:56:06.176619Z

;; a simple global instance, will probably be removed
(def S
  (try
    ;; We cannot run the simple-supervisor thread in a static context inside
    ;; native image.
    (dummy-supervisor)
    #_#?(:clj (if (native-image-build?)
                (dummy-supervisor)
                (simple-supervisor))
         :cljs (simple-supervisor))
    (catch #?(:clj Exception :cljs js/Error) _
      (dummy-supervisor) #_(simple-supervisor))))

borkdude 2022-05-19T08:56:23.922559Z

yes, but it will still be used simply because it's a top level thing

grounded_sage 2022-05-19T08:56:29.659499Z

I commented it out to just use the dummy-supervisor which was added for the native image support

borkdude 2022-05-19T08:56:37.477409Z

ah right

borkdude 2022-05-19T08:57:15.578879Z

maybe you can comment out more until you don't get the error and then detect where the thread is being started

๐Ÿ‘ 1
grounded_sage 2022-05-19T08:59:01.481019Z

Yea I was just trying to make it work quickly because working on getting it right isn't a major priority atm. So was hoping I could just initialise the class at runtime.

borkdude 2022-05-19T08:59:39.465789Z

anything used from clojure cannot be initialized at runtime, unless you push it to a function (or delay which is essentially a function)

grounded_sage 2022-05-19T09:00:42.511269Z

okay thats a valuable insight. Thanks. Perhaps I will try and pick apart superv.async a bit more and find where this issue is.