Fork me on GitHub
#graalvm
<
2019-12-04
>
sogaiu13:12:45

@borkdude why did you close https://github.com/lread/clj-graal-docs/issues/4 - i am trying to figure out why a particular project can be built (and run) using your uberjar method, but fails when using clj.native-image

borkdude15:12:28

@sogaiu Now the observed difference is gone in the hello world version of that repo

borkdude15:12:41

I don't exactly remember the details, it was just a temporary issue for something I was figuring out

borkdude15:12:54

one issue I've had with clj-native was the order in which things were compiled or that not all the source was compiled

borkdude15:12:00

so that could be your problem

sogaiu15:12:21

thanks for the hints -- my quest continues 🙂

borkdude15:12:04

what is the error you're getting

sogaiu17:12:18

@borkdude i get multiple of this type of thing:

Error: Unsupported features in 12 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: clojure.tools.reader.reader_types.IPushbackReader. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace: 
	at parsing rewrite_clj.reader$unread.invokeStatic(reader.clj:115)
Call path from entry point to rewrite_clj.reader$unread.invokeStatic(Object, Object): 
	at rewrite_clj.reader$unread.invokeStatic(reader.clj:112)
	at rewrite_clj.reader$unread.invoke(reader.clj:112)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at script.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
full output (from command invocation) at: https://pastebin.com/UM8ty9x6

borkdude17:12:23

@sogaiu can you compare the .class files in the uberjar to the .class files produced by the clj native tool?

borkdude17:12:35

if there's a difference, it's probably that

sogaiu17:12:28

i'll take a look (iiuc IPushbackReader comes into existence via defprotocol -- is that something that can be understood by native-image at build time?)

borkdude17:12:48

I think so yes, as long as the protocol is defined at compile time

borkdude17:12:56

I'm using this in babashka as well

sogaiu17:12:22

ok, thanks

borkdude17:12:37

top level I mean, defined during the native image analysis

sogaiu17:12:47

so far in the comparison, it appears that via lein i get some class files (as well as the source) for tools.reader.reader_types, but i don't see anything in the classes directory for clj-native-image -- that looks relevant < @taylor

sogaiu17:12:59

the weird thing is that this used to work

borkdude17:12:44

I've had a similar problem, that's why I went to lein eventually, since that has more predictable compilation

borkdude17:12:52

btw, I also had a script somewhere to compile all namespaces

borkdude17:12:11

let me see if I can find it, but that will be later, since I'm cooking dinner right now

borkdude17:12:41

fwiw I can't find it anymore, I think I deleted it while making a hello world for clj-graal-docs and I didn't need it anymore

borkdude17:12:56

but it used tools.namespace to list all namespaces

sogaiu17:12:56

np -- thanks for considering 🙂

borkdude17:12:05

and then ran compile on it manually

sogaiu12:12:27

i did some more experiments and i think i more or less confirmed what you said -- may be you saw already, but fwiw, notes are at: https://github.com/taylorwood/clj.native-image/issues/20#issuecomment-561976573

borkdude12:12:32

I saw it, thanks

sogaiu18:12:16

it seems odd because the part of tools.namespace that clj.native-image is using is just find-namespaces-in-dir -- i've compared the values for both versions of tools.namespace (the one from the version of clj.native-image that works from the one that doesn't) and they return the same value.

taylor23:12:50

thanks for looking into this! I’ll try to repro/figure it out soon, otherwise I may rollback to the previous version of that dep as a workaround :man-shrugging:

sogaiu23:12:56

fwiw, i filed a related issue with some more details

taylor23:12:56

yeah I’m looking at what changed between those versions

taylor23:12:04

> Release 1.0.0-alpha1 on 02 Nov 2015 > Reader types implement Closeable > Added pushback-reader > Improved interoperability between reader types and java readers > Release 0.10.0 on 09 Oct 2015 > Better error for invalid octal literal

sogaiu23:12:22

fwiw, if i explicitly set my project to use tools.reader 1.3.2 and (compile 'script), at least some (possibly all?) relevant reader_types .class files appear within the classes directory

sogaiu00:12:28

i tried adding clj.native-image to my deps.edn in the :deps section. from a fresh repl, if the first thing i do is (compile 'script), i see class files for reader_types from a fresh repl, if i (require '[clj.native-image]) and then (compile 'script), i don't get the reader_types class files

sogaiu00:12:21

and this is with my deps.edn file explicitly asking for tools.reader 1.3.2

sogaiu00:12:42

also tried adding java.classpath 0.3.0 and tools.namespace 0.3.1 to my deps.edn -- same result

sogaiu00:12:19

if i (require 'clojure.tools.namespace.find) before (compile 'script), the reader_type class files don't appear

sogaiu00:12:45

ah, could it be that things that are required end up being cached so they aren't going to be written to disk?

sogaiu00:12:19

@taylor TIL from noisesmith about *loading-verbosely* -- when i do:

(in-ns 'clojure.core)
(binding [*loading-verbosely* true] (compile 'script))
i see what is loaded during compilation. if i require clojure.tools.namespace.find before compilation, reader-types doesn't appear to get loaded during compile (perhaps it is cached?) if i don't require anything first and do compile while observing what's loaded, i do see reader-types

sogaiu00:12:52

so, possibly using mranderson or similar to "vendor" clj.native-image's dependencies might actually address this situation -- may be there are other alternatives

sogaiu01:12:44

don't know if there's a way to force compile to load everything necessary from disk instead of using cached info

sogaiu01:12:26

there's a *loaded-libs* in clojure.core -- may be this is a factor in deciding whether to load something. i don't know if it's a good idea, but may be it can be modified to not contain certain libs to force them to get loaded during a call to compile

sogaiu01:12:00

as an experiment at the repl, i tried (dosync (ref-set *loaded-libs* ...)) after a require of clj.native-image (i saved the value of *loaded-libs* before the require and used that in ref-set). this seemed to affect compile's behavior appropriately. *loaded-libs* is private though and this approach doesn't seem right.

sogaiu02:12:44

explicitly compiling each ns might work

taylor00:12:24

hey just wanted to say thanks so much for the detailed research/notes for these issues, and apologize that I won’t have much time until after holidays to look at it 😞

sogaiu00:12:35

no worries! thanks for the update 🙂

sogaiu18:12:34

there is only one value in both cases, and it is script

sogaiu18:12:46

so iiuc, there is only one invocation of compile

sogaiu18:12:08

(let [deps-map (merged-deps)
      namespaces (mapcat (comp find-namespaces-in-dir io/file) (:paths deps-map))]
  (prep-compile-path)
  (doseq [ns (distinct (cons main-ns namespaces))]
    (println "Compiling" ns)
    (compile (symbol ns)))

heikki.hamalainen20:12:54

What is the current situation of making native images which use AWS SDK? It seems that Amazonica seems to not work but have anyone tried Cognitects AWS SDK?

taylor23:12:00

I think I tried using S3 with Cognitect’s AWS API several months ago and hit some snags, but can’t remember details

borkdude20:12:45

why does it not work? reflection?

borkdude20:12:56

if it's that, you can make it work using a reflection config