Fork me on GitHub
#graalvm-mobile
<
2021-06-25
>
borkdude08:06:46

@smith.adriane @raspasov It's worth trying to not use --initialize-at-build-time but using this approach: https://github.com/oracle/graal/discussions/3476#discussioncomment-897705 to see if this helps.

👍 2
phronmophobic17:06:32

I'm trying it out. Just blindly following the example produces the following errors:

To see how the classes got initialized, use --trace-class-initialization=primitive_math$variadic_proxy,primitive_math$using_primitive_operators_QMARK_,primitive_math$use_primitive_operators,primitive_math$variadic_predicate_proxy,primitive_math$unuse_primitive_operators
[mobiletest-uber:95490]     analysis:  40,021.43 ms,  3.22 GB
Error: Classes that should be initialized at run time got initialized during image building:
 primitive_math$variadic_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_proxy got initialized use --trace-class-initialization=primitive_math$variadic_proxy
primitive_math$using_primitive_operators_QMARK_ was unintentionally initialized at build time. To see why primitive_math$using_primitive_operators_QMARK_ got initialized use --trace-class-initialization=primitive_math$using_primitive_operators_QMARK_
primitive_math$use_primitive_operators was unintentionally initialized at build time. To see why primitive_math$use_primitive_operators got initialized use --trace-class-initialization=primitive_math$use_primitive_operators
primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_predicate_proxy got initialized use --trace-class-initialization=primitive_math$variadic_predicate_proxy
primitive_math$unuse_primitive_operators was unintentionally initialized at build time. To see why primitive_math$unuse_primitive_operators got initialized use --trace-class-initialization=primitive_math$unuse_primitive_operators

com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
 primitive_math$variadic_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_proxy got initialized use --trace-class-initialization=primitive_math$variadic_proxy
primitive_math$using_primitive_operators_QMARK_ was unintentionally initialized at build time. To see why primitive_math$using_primitive_operators_QMARK_ got initialized use --trace-class-initialization=primitive_math$using_primitive_operators_QMARK_
primitive_math$use_primitive_operators was unintentionally initialized at build time. To see why primitive_math$use_primitive_operators got initialized use --trace-class-initialization=primitive_math$use_primitive_operators
primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_predicate_proxy got initialized use --trace-class-initialization=primitive_math$variadic_predicate_proxy
primitive_math$unuse_primitive_operators was unintentionally initialized at build time. To see why primitive_math$unuse_primitive_operators got initialized use --trace-class-initialization=primitive_math$unuse_primitive_operators
I'll try to use --trace-class-initialization and see if I can get a working list

borkdude17:06:15

@smith.adriane you'll need to use the namespace list of your specific application, generated using the sniippet I provided

borkdude17:06:23

were you doing that?

borkdude17:06:46

it seems you'll need to add primitive_math

borkdude17:06:04

perhaps the snippet didn't account for some characters

phronmophobic17:06:22

just noticed that your list starts with "clojure", but it doesn't show up in mine

borkdude17:06:44

I did a (cons "clojure" ..) somewhere manually

borkdude17:06:08

(->> (map ns-name (all-ns)) (remove #(str/starts-with? % "clojure")) (map #(str/split (str %) #"\.")) (keep butlast) (map #(str/join "." %)) distinct (map munge) (cons "clojure"))
see the end

borkdude17:06:14

you'll first need to require your main namespace

phronmophobic17:06:28

whoops. I think just forgot how to read for a second

phronmophobic17:06:36

trying with "primitive_math" included

borkdude17:06:25

execute the above in a REPL, but first do:

(require 'your.main)
(require '[clojure.string :as str])

phronmophobic17:06:47

I added the following function:

(defn initialize-at-build-time-list [& args]
  (println
   (->> (map ns-name (all-ns))
        (remove #(clojure.string/starts-with? % "clojure"))
        (map #(clojure.string/split (str %) #"\."))
        (keep butlast)
        (map #(clojure.string/join "." %))
        distinct
        (map munge)
        (cons "clojure")
        (clojure.string/join ","))))
so I can have it referenced in my compile script:
INITIALIZE_AT_BUILD_TIME=`clojure -X com.phronmophobic.mobiletest/initialize-at-build-time-list
` ...
--initialize-at-build-time="$INITIALIZE_AT_BUILD_TIME" \

borkdude17:06:18

yeah cool. can you also echo what's in the var?

phronmophobic17:06:41

set -x will print it as its executed

borkdude17:06:51

I mean, can you list it here?

phronmophobic17:06:54

+ INITIALIZE_AT_BUILD_TIME=clojure,tech.v3.datatype,sci.impl,babashka.nrepl,tech.v3.resource,bencode,flatland.ordered,com.phronmophobic,babashka.nrepl.impl,datascript,edamame.impl,tech.v3,tech.v3.parallel,sci.addons,tech.v3.datatype.ffi,sci,primitive_math

borkdude17:06:22

that seems good, but I don't see any of your namespaces...

phronmophobic17:06:32

com.phronmophobic

borkdude17:06:45

ok, so this should do it then... hopefully

phronmophobic17:06:11

primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. com.phronmophobic.mobiletest caused initialization of this class with the following trace: 
	at primitive_math$variadic_predicate_proxy.<clinit>(primitive_math.clj:27)

phronmophobic17:06:37

does order matter?

borkdude17:06:46

no. what is primitive_math?

borkdude17:06:02

I think the issue here may be that it's not in a package, just a bare class name

phronmophobic17:06:15

It's used by dtype-next

borkdude17:06:21

this is why you should probably never use a single segment namespace

phronmophobic17:06:10

wasn't me! 😛

borkdude17:06:11

I'll take this up with graalvm devs

borkdude17:06:38

you'll probably run into lots of classes from this library that you should explicitly list...

borkdude17:06:45

because it has no package

phronmophobic17:06:35

is it fixable without graalvm changes?

phronmophobic17:06:09

just by adding primitive_math$variadic_predicate_proxy and whatever else it complains about?

borkdude17:06:59

but you'll likely need to enumerate quite few of them..

borkdude17:06:58

you could perhaps do this using the list of class files in the classes/primitive_math directory or so

phronmophobic17:06:28

I've added all the classes that were printed

borkdude17:06:40

it will likely come up with a new class

phronmophobic17:06:53

if only they returned the error as edn 😛 (or even json).

phronmophobic17:06:36

{:unintentially-initialized ["primitive_math$variadic_predicate_proxy" ...]
 :error "Classes that should be initialized at run time got initialized during image building"}

borkdude17:06:09

escape the dollar?

phronmophobic17:06:47

the snippet was a wishful example of native image output

phronmophobic17:06:50

it seems like it's compiling

phronmophobic17:06:32

rather than using:

(defn initialize-at-build-time-list [& args]
  (println
   (->> (map ns-name (all-ns))
        (remove #(clojure.string/starts-with? % "clojure"))
        (map #(clojure.string/split (str %) #"\."))
        (keep butlast)
        (map #(clojure.string/join "." %))
        distinct
        (map munge)
        (cons "clojure")
        (clojure.string/join ","))))
It seems like it would be more robust to just examine the classes in the uberjar. Would that work?

borkdude17:06:46

@smith.adriane well, the point in the linked discussion is that not all classes should be initialized at build time, only the clojure ones

borkdude17:06:18

so if there are other Java classes in your jar then those should not be listed preferably

borkdude17:06:29

since native-image should figure it out by itself

phronmophobic17:06:45

oh, interesting

borkdude17:06:27

and the deadlock can happen because of initializing at build time for some of the Java stuff I think

borkdude17:06:03

but it remains to be seen if the above approach has really solved the issue. did you push the change? I could try it locally

phronmophobic17:06:33

Will do. one sec

borkdude17:06:14

I will also try this approach for https://github.com/borkdude/tools-deps-native-experiment which has a similar issue

phronmophobic17:06:43

ok, pushed the changes

phronmophobic17:06:05

It seems like it should be possible to generate the list of clojure classes since projects like depstar need to produce a list for AOT compilation

borkdude17:06:23

good thinking

phronmophobic19:06:29

Using the package for initialize at build time can have false positive since it's possible to have a java class that shares the same package as clojure code. I tried filtering all classnames in an uber jar by matching them with namespaces:

(defn list-resources [path]
  (let [jar (java.util.jar.JarFile. path)  
        entries (.entries jar)]
    (loop [result  []]
      (if (.hasMoreElements entries)
        (recur (conj result (.. entries nextElement getName)))
        result))))

(defn namespace->namespace-key [ns]
  (-> ns
      ns-name
      str
      (clojure.string/split #"\.")
      (->> (map munge))))

(defn class-path->namespace-key [fname]
  (-> fname
      (clojure.string/replace #"(\$[^.]+)?.class$" "")
      (clojure.string/split #"/"))
  )

(defn class-path->classname [fname]
  (-> (subs fname 0 (- (count fname)
                       (count ".class")))
      (clojure.string/replace #"/" ".")))

(defn clojure-classes-in-jar [jar-path]
  (let [resources (list-resources jar-path)
        ns-keys (into
                 #{}
                 (map namespace->namespace-key)
                 (all-ns))
        classes (into #{}
                      (comp
                       (filter #(clojure.string/ends-with? % ".class"))
                       (filter (fn [class-path]
                                 (contains? ns-keys (class-path->namespace-key class-path))))
                       (map class-path->classname))
                      resources)]
    classes))

(comment
  (spit "initialize-at-buildtime-classes.txt"
        (clojure.string/join ","
                             (clojure-classes-in-jar  "target/mobiletest-uber.jar")))

  ;; INITIALIZE_AT_BUILD_TIME=$(cat initialize-at-buildtime-classes.txt)
  ;;     --initialize-at-build-time=clojure,"$INITIALIZE_AT_BUILD_TIME" \
)
This doesn't quite work because it doesn't include classes created by defrecord. It's also a comically large list.

phronmophobic19:06:46

However it should be possible to modify the above code to include defrecord classes if including java classes that share packages with clojure namespaces isn't a big deal

borkdude19:06:18

> if including java classes that share packages with clojure namespaces isn't a big deal sometimes it is, in the case of httpkit for example

borkdude19:06:22

see its README

borkdude19:06:33

but that would also be the case with the namespace approach

phronmophobic19:06:35

I guess it should also be possible to list all classes created by defrecord and include them when filtering classes in an uberjar