Fork me on GitHub
#graalvm
<
2019-11-28
>
sogaiu03:11:13

@taylor i just noticed that one of my projects no longer builds with the changes in this commit: https://github.com/taylorwood/clj.native-image/commit/b3823a48be75122b9671c86ce5353a85589ef15f (it does build with the commit right before it) the output from native-image looks like this:

Error: Unsupported features in 12 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: clojure.tools.reader.reader_types.IPushbackReader. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace: 
	at parsing rewrite_clj.reader$unread.invokeStatic(reader.clj:115)
Call path from entry point to rewrite_clj.reader$unread.invokeStatic(Object, Object): 
	at rewrite_clj.reader$unread.invokeStatic(reader.clj:112)
	at rewrite_clj.reader$unread.invoke(reader.clj:112)
	at clojure.tools.reader.default_data_readers.proxy$java.lang.ThreadLocal$ff19274a.equals(Unknown Source)
	at java.util.HashMap.getNode(HashMap.java:579)
	at java.util.HashMap.get(HashMap.java:557)
	at com.oracle.svm.jni.access.JNIReflectionDictionary.getClassObjectByName(JNIReflectionDictionary.java:128)
	at com.oracle.svm.jni.functions.JNIFunctions.FindClass(JNIFunctions.java:314)
	at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_FindClass_3ec1032c6cb9443725d1e68194130533bfb04076(generated:0)

sogaiu03:11:02

i tried what you mentioned in: https://github.com/taylorwood/clj.native-image/issues/15#issuecomment-550573681 , but got a result similar to what you got.

sogaiu03:11:22

i had "--initialize-at-build-time" before (build unsuccessful) and changed this to "--initialize-at-build-time=clojure"

taylor20:11:10

Thanks so much for these detailed notes!

taylor20:11:10

I’m really interested about the thing where upgrading that dependency introduced a problem

sogaiu21:11:41

hmm, not quite sure what you mean about upgrading tools.deps introducing a problem. are you referring to: https://github.com/taylorwood/clj.native-image/commit/b3823a48be75122b9671c86ce5353a85589ef15f ? atm, it looks to me like it's tools.namespace that is a culprit.

taylor01:11:48

Sorry, misread

taylor01:11:05

But yes, that upgrade

sogaiu01:11:48

i have toggled the version back and forth and tried building with just the version changed -- it appears reproducible here, as in, with the newer version i get errors, with the older version it's fine

sogaiu01:11:25

the project i tried with apart from fc4, is: https://github.com/sogaiu/adorn -- i'm not sure if that one will work for other folks because one of its dependencies may have been broken within the last day, but perhaps it's worth a try?

sogaiu16:11:10

i compared the -imagecp values for the 2 runs of building adorn and found some differences. tn 0.2.11 (build succeeds), has:

tools.namespace-0.2.11.jar
tools.reader-1.2.2.jar
tn 0.3.1 (build fails), has:
tools.namespace-0.3.1.jar
java.classpath-0.3.0.jar 
tools.reader-1.3.2.jar

sogaiu16:11:47

i checked the value of:

(mapcat (comp find-namespaces-in-dir io/file) (:paths deps-map))
and in both runs it was:
(script)
so may be it's not like there was change in the behavior of find-namespaces-in-dir

sogaiu16:11:00

does this help at all?

👍 4
sogaiu17:11:47

i also checked the 2 runs of building fc4 and found the following differences among the -imagecp values: tn 0.2.11, has:

tools.namespace-0.2.11.jar
tools.reader-1.0.0-beta4.jar
tn 0.3.1, has:
tools.namespace-0.3.1.jar
java.classpath-0.3.0.jar 
tools.reader-1.3.2.jar

sogaiu17:11:17

the values for: (mapcat (comp find-namespaces-in-dir io/file) (:paths deps-map)) were identical:

(fc4.files fc4.image-utils fc4.integrations.structurizr.express.chromium-renderer fc4.integrations.structurizr.express.export fc4.integrations.structurizr.express.format fc4.integrations.structurizr.express.snap fc4.integrations.structurizr.express.spec fc4.integrations.structurizr.express.yaml .cli.main .cli.util .dsl .render .util fc4.io.watch .yaml fc4.model fc4.rendering fc4.spec fc4.styles fc4.util fc4.view fc4.yaml)

taylor22:11:21

yeah that’s the first thing that came to my mind is checking to see if the calculated classpaths were different :thinking_face:

sogaiu22:11:42

the type of error i'm seeing is:

Error: Unsupported features in 12 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: clojure.tools.reader.reader_types.IPushbackReader. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace: 
	at parsing rewrite_clj.reader$unread.invokeStatic(reader.clj:115)
Call path from entry point to rewrite_clj.reader$unread.invokeStatic(Object, Object): 
	at rewrite_clj.reader$unread.invokeStatic(reader.clj:112)
	at rewrite_clj.reader$unread.invoke(reader.clj:112)
	at clojure.tools.reader.default_data_readers.proxy$java.lang.ThreadLocal$ff19274a.equals(Unknown Source)
	at java.util.HashMap.getNode(HashMap.java:579)
	at java.util.HashMap.get(HashMap.java:557)
	at com.oracle.svm.jni.access.JNIReflectionDictionary.getFieldNameByID(JNIReflectionDictionary.java:278)
	at com.oracle.svm.jni.functions.JNIFunctions.ToReflectedField(JNIFunctions.java:836)
	at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_ToReflectedField_80d8233579d5215df0227b770e5c01228a0de9b9(generated:0)

sogaiu22:11:28

building with --allow-incomplete-classpath succeeds -- startup of the resulting binary is successful without a crash, but when i try to use it for what it's supposed to do, it crashes with:

Exception in thread "main" java.lang.NoClassDefFoundError: clojure.tools.reader.reader_types.IndexingReader
	at rewrite_clj.reader$position.invokeStatic(reader.clj:125)
	at rewrite_clj.reader$position.invoke(reader.clj:122)
	at rewrite_clj.reader$read_with_meta.invokeStatic(reader.clj:131)
	at rewrite_clj.reader$read_with_meta.invoke(reader.clj:128)
	at rewrite_clj.parser.core$parse_next.invokeStatic(core.clj:36)
	at rewrite_clj.parser.core$parse_next.invoke(core.clj:34)
	at rewrite_clj.parser$parse.invokeStatic(parser.clj:13)
	at rewrite_clj.parser$parse.invoke(parser.clj:10)
	at rewrite_clj.parser$parse_all$fn__2947.invoke(parser.clj:18)
	at clojure.core$repeatedly$fn__6176.invoke(core.clj:5089)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:528)
	at clojure.core$seq__5124.invokeStatic(core.clj:137)
	at clojure.core$take_while$fn__5638.invoke(core.clj:2896)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:528)
	at clojure.core$seq__5124.invokeStatic(core.clj:137)
	at clojure.core$dorun.invokeStatic(core.clj:3125)
	at clojure.core$doall.invokeStatic(core.clj:3140)
	at clojure.core$doall.invoke(core.clj:3140)
	at rewrite_clj.parser$parse_all.invokeStatic(parser.clj:20)
	at rewrite_clj.parser$parse_all.invoke(parser.clj:15)
	at rewrite_clj.parser$parse_string_all.invokeStatic(parser.clj:35)
	at rewrite_clj.parser$parse_string_all.invoke(parser.clj:32)
	at rewrite_clj.zip.base$of_string.invokeStatic(base.clj:73)
	at rewrite_clj.zip.base$of_string.invoke(base.clj:69)
	at script$find_defn_zloc_at.invokeStatic(script.clj:12)
	at script$find_defn_zloc_at.invoke(script.clj:10)
	at script$_main.invokeStatic(script.clj:37)
	at script$_main.doInvoke(script.clj:27)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at script.main(Unknown Source)

sogaiu22:11:24

i tried adding clojure.tools.reader.reader_types.IndexingReader to reflection.json but during a subsequent native-image build i see:

WARNING: Could not resolve clojure.tools.reader.reader_types.IndexingReader for reflection configuration.

sogaiu22:11:09

...and still get the exception during execution

sogaiu22:12:58

@taylor oddly enough if i use lein to build an uberjar first, native-image doesn't appear to have any issues

beders23:05:10

3 years later, I’m hitting the same thing. And I guess it must be a classpath issue. The classpath running

clojure -M -e "(System/getProperty \"java.class.path\")"   
is very different from the one running clj -Anative-image The second one has a shit ton more classes and - curiously - if I compile my main namespace with
clojure -M -e "(compile ')"
I see all the clojure.tools.reader.reader_types classes compiled. If I let clj.native do the compile, they are not there. As I’ve pointed out a few times: tools.deps is not a stable build tool. The classpath order is not defined and the second classpath might have an older version of clojure.tools.reader packaged up somewhere and that’s why the build breaks.

👍 1
beders23:05:28

in that case clj.native-image requires tools.namespace 1.0.0 which in turn requires an older version of clojure.tools.reader

jaihindhreddy12:11:09

I am a complete Graal neophyte, and I keep hearing stuff like "Graal is based on a closed world assumption". Here's what I'm wondering: 1. How does this tie into the dynamic nature of Clojure? 2. Which subset of Clojure's features and/or constructs are compatible with Graal? 3. Do the answers to the above two questions depend on whether we're talking about GraalVM HotSpot mode vs native image?

taylor20:11:16

The closed world thing only applies to AOT compiling your program to a native image

taylor20:11:51

Otherwise you can use the GraalVM basically as you would a JVM

taylor20:11:41

Compiling stuff to a native image does prohibit a lot of dynamic runtime stuff you could otherwise do with Clojure

borkdude13:11:20

@jaihindhreddy I'm only familiar with native image. Closed world means that it analyses reachable code at compile time and will strip everything not needed.

borkdude13:11:30

Kind of like the Google Closure compiler for CLJS + eval in CLJS

borkdude13:11:26

See https://github.com/lread/clj-graal-docs/blob/master/doc/external-resources.md for a list of projects that are currently working in graal

borkdude13:11:38

@lvh Did you end up announcing your blogpost somewhere? Maybe add it to ^