Fork me on GitHub
#graalvm
<
2021-06-11
>
Karol Wójcik06:06:21

Is it possible to specify list of reflection entries which should be ignored when generating native-configurations using GraalVM agent? It's makes me crazy that I have to each time manually trim some of the entries. PS: I'm looking for automated way, so that I can built it into #holy-lambda

borkdude06:06:28

@karol.wojcik there is something even better that I’m going to try out today. You can ignore the place where the reflection happened, so if you specify the place in the Clojure compiler it should only leave the real problematic cases

👍 3
Karol Wójcik06:06:47

Can wait to see the progress in that area. Please notify me when you make any progress. I'm eager to help with this!

borkdude06:06:28

But you can also have access based filters

borkdude08:06:45

@karol.wojcik This filter works reasonably well:

borkdude@MBP2019 ~/Dropbox/temp/refl $ cat filter.json
{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat src/refl/main.clj
(ns refl.main
  (:gen-class))

(defn refl-str [s]
  (.length s))

(defn -main [& _]
  (println (refl-str "foo")))

borkdude@MBP2019 ~/Dropbox/temp/refl $ java -agentlib:native-image-agent=caller-filter-file=filter.json,config-output-dir=. -cp $(clojure -Spath):classes refl.main
3
borkdude@MBP2019 ~/Dropbox/temp/refl $ cat reflect-config.json
[
{
  "name":"java.lang.String",
  "allPublicMethods":true
},
{
  "name":"java.lang.reflect.Method",
  "methods":[{"name":"canAccess","parameterTypes":["java.lang.Object"] }]
},
{
  "name":"java.util.Properties",
  "allPublicMethods":true
}
]

Karol Wójcik08:06:55

I'll take a look how it behaves for some more complex cases. Probably we would need some kind of filtering for resources as well.

borkdude08:06:01

I'm mostly interested in the reflection config though

borkdude08:06:12

not sure where the java.util.Properties thing comes from though, but it's only one thing

borkdude08:06:50

I made an issue at graal here to make this process a bit easier: https://github.com/oracle/graal/issues/3461

borkdude08:06:48

ah, this is probably already supported! trace-output=foo

borkdude08:06:23

and indeed it logs:

{"caller_class":"clojure.lang.Reflector", "args":[], "function":"getMethods", "tracer":"reflect", "class":"java.util.Properties"}
Not sure why Clojure does this reflection but ok

borkdude08:06:33

@karol.wojcik So using this trace-output option I found out you can get the same config using only:

{ "rules": [
  {"excludeClasses": "clojure.lang.RT"}
]
}

🤯 3
❤️ 6
borkdude08:06:40

That's it :)

Karol Wójcik08:06:51

@borkdude You must be kidding me! 😄

borkdude09:06:15

I would just leave out `{ "name":"java.util.Properties", "allPublicMethods":true } ` though, as this class pulls in XML libraries and bloats your image. I'm not sure why Clojure does this reflection during the lifecycle of a program, but I guess that's for another time.

borkdude09:06:34

but this can be configured using an access filter as well

borkdude09:06:34

Hm, this config is better it turns out:

{ "rules": [
  {"excludeClasses": "clojure.**"},
  {"includeClasses": "clojure.lang.Reflector"}
]
}

Karol Wójcik13:06:08

It's not a silver bullet for everything it seems. For instance compiling project with cognitect.aws lacks some of the reflections:

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection, :cognitect.http-client/throwable #error {
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :cause Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :via
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [{:type java.lang.IllegalArgumentException
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :message Class java.nio.HeapByteBuffer[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :at [com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]}]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 :trace
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [[com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets arrayHubErrorStub SubstrateAllocationSnippets.java 295]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1754]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.RT seqToTypedArray RT.java 1750]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core$into_array invokeStatic core.clj 3440]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$map__GT_jetty_request invokeStatic http_client.clj 108]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client$fn__10581 invoke http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client.Client submit_STAR_ http_client.clj 226]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.http_client$submit invokeStatic http_client.clj 213]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http.cognitect$create$reify__10605 _submit cognitect.clj 10]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.http$submit invokeStatic http.clj 45]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206$fn__10220 invoke client.clj 97]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204$fn__10206 invoke client.clj 96]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177$state_machine__5488__auto____10204 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic ioc_macros.clj 978]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic ioc_macros.clj 980]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [cognitect.aws.client$send_request$fn__10177 invoke client.clj 84]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1149]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 624]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.core.async.impl.concurrent$counted_thread_factory$reify__228$fn__229 invoke concurrent.clj 29]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [clojure.lang.AFn run AFn.java 22]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [java.lang.Thread run Thread.java 748]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.thread.JavaThreads threadStartRoutine JavaThreads.java 553]
2021/06/11/[$LATEST]02d90d8b74584141bf50e20a0493089b 2021-06-11T13:30:38.265000 [com.oracle.svm.core.posix.thread.PosixJavaThreads pthreadStartRoutine PosixJavaThreads.java 192]]}}

borkdude13:06:39

yeah, I think this is coming from their http client

borkdude13:06:44

or something

borkdude13:06:49

so the agent didn't catch this?

borkdude13:06:07

ah well, it's not perfect then, but it can help

borkdude13:06:14

that's how I always approached it

Karol Wójcik13:06:52

It's coming from RT

borkdude13:06:59

clojure.lang.RT is known to use reflection around arrays

borkdude13:06:12

this is a known issue, but since we ignored it, you don't get to see it

borkdude13:06:50

@karol.wojcik I think we can get around this by building on the output of the trace-output instead. E.g. for array creation you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["[Ljava.lang.String;"], "function":"newInstance", "tracer":"reflect", "class":"java.lang.reflect.Array"}

borkdude13:06:12

but for "false positives" you see this:

{"caller_class":"clojure.lang.RT", "result":true, "args":["refl.main__init"], "function":"forName", "tracer":"reflect", "class":"java.lang.Class"}

borkdude13:06:19

so that's an easy filter operation

borkdude13:06:53

or perhaps combine the info

Karol Wójcik14:06:22

I don't follow: > by building on the output of the `trace-output` instead What do you mean by that? Can I filter trace-output and feed GraalVM with it to generate reflect-config?

borkdude14:06:04

I'll get back to this later, meeting

Karol Wójcik10:06:35

Huh. I read GraalVM configuration once again. I know what you mean now. I will try to implement this filtering! Thanks @borkdude

borkdude10:06:47

I'm also experimenting with it

❤️ 3
Karol Wójcik13:06:16

I will try it out tomorrow morning! Getting ready for some party 🎈. Wish you a great day @borkdude !

borkdude11:06:26

I think I found the spot where the false positive java.util.Properties reflection comes from. https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core.clj#L7085 Would be nice if this could be resolved, not sure if the core team is open for this change, because for Clojure itself it's not that important.

👍 3
Karol Wójcik11:06:05

Looks quite easy to fix. I'll try to make a patch for it

borkdude11:06:32

I think the fix is not the issue, it's more if it will be accepted or not ;)

😄 3
borkdude12:06:56

ok, I'll make a JIRA + patch once I confirm locally that this solves the "false positive" in the graal generated config

👍 3
borkdude13:06:46

issue + patch: https://clojure.atlassian.net/browse/CLJ-2636 confirmed that this gets rid of the "false positive" in the generated graal config