Clojurians Log v2

@smith.adriane @raspasov It's worth trying to not use --initialize-at-build-time but using this approach: https://github.com/oracle/graal/discussions/3476#discussioncomment-897705 to see if this helps.

I'm trying it out. Just blindly following the example produces the following errors:

To see how the classes got initialized, use --trace-class-initialization=primitive_math$variadic_proxy,primitive_math$using_primitive_operators_QMARK_,primitive_math$use_primitive_operators,primitive_math$variadic_predicate_proxy,primitive_math$unuse_primitive_operators
[mobiletest-uber:95490]     analysis:  40,021.43 ms,  3.22 GB
Error: Classes that should be initialized at run time got initialized during image building:
 primitive_math$variadic_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_proxy got initialized use --trace-class-initialization=primitive_math$variadic_proxy
primitive_math$using_primitive_operators_QMARK_ was unintentionally initialized at build time. To see why primitive_math$using_primitive_operators_QMARK_ got initialized use --trace-class-initialization=primitive_math$using_primitive_operators_QMARK_
primitive_math$use_primitive_operators was unintentionally initialized at build time. To see why primitive_math$use_primitive_operators got initialized use --trace-class-initialization=primitive_math$use_primitive_operators
primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_predicate_proxy got initialized use --trace-class-initialization=primitive_math$variadic_predicate_proxy
primitive_math$unuse_primitive_operators was unintentionally initialized at build time. To see why primitive_math$unuse_primitive_operators got initialized use --trace-class-initialization=primitive_math$unuse_primitive_operators

com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
 primitive_math$variadic_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_proxy got initialized use --trace-class-initialization=primitive_math$variadic_proxy
primitive_math$using_primitive_operators_QMARK_ was unintentionally initialized at build time. To see why primitive_math$using_primitive_operators_QMARK_ got initialized use --trace-class-initialization=primitive_math$using_primitive_operators_QMARK_
primitive_math$use_primitive_operators was unintentionally initialized at build time. To see why primitive_math$use_primitive_operators got initialized use --trace-class-initialization=primitive_math$use_primitive_operators
primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. To see why primitive_math$variadic_predicate_proxy got initialized use --trace-class-initialization=primitive_math$variadic_predicate_proxy
primitive_math$unuse_primitive_operators was unintentionally initialized at build time. To see why primitive_math$unuse_primitive_operators got initialized use --trace-class-initialization=primitive_math$unuse_primitive_operators

@smith.adriane you'll need to use the namespace list of your specific application, generated using the sniippet I provided

were you doing that?

yea

it seems you'll need to add primitive_math

perhaps the snippet didn't account for some characters

just noticed that your list starts with "clojure", but it doesn't show up in mine

I did a (cons "clojure" ..) somewhere manually

(->> (map ns-name (all-ns)) (remove #(str/starts-with? % "clojure")) (map #(str/split (str %) #"\.")) (keep butlast) (map #(str/join "." %)) distinct (map munge) (cons "clojure"))

you'll first need to require your main namespace

whoops. I think just forgot how to read for a second

trying with "primitive_math" included

execute the above in a REPL, but first do:

(require 'your.main)
(require '[clojure.string :as str])

I added the following function:

(defn initialize-at-build-time-list [& args]
  (println
   (->> (map ns-name (all-ns))
        (remove #(clojure.string/starts-with? % "clojure"))
        (map #(clojure.string/split (str %) #"\."))
        (keep butlast)
        (map #(clojure.string/join "." %))
        distinct
        (map munge)
        (cons "clojure")
        (clojure.string/join ","))))

INITIALIZE_AT_BUILD_TIME=`clojure -X com.phronmophobic.mobiletest/initialize-at-build-time-list

--initialize-at-build-time="$INITIALIZE_AT_BUILD_TIME" \

yeah cool. can you also echo what's in the var?

env var

set -x will print it as its executed

I mean, can you list it here?

+ INITIALIZE_AT_BUILD_TIME=clojure,tech.v3.datatype,sci.impl,babashka.nrepl,tech.v3.resource,bencode,flatland.ordered,com.phronmophobic,babashka.nrepl.impl,datascript,edamame.impl,tech.v3,tech.v3.parallel,sci.addons,tech.v3.datatype.ffi,sci,primitive_math

that seems good, but I don't see any of your namespaces...

right?

com.phronmophobic

ah right

ok, so this should do it then... hopefully

primitive_math$variadic_predicate_proxy was unintentionally initialized at build time. com.phronmophobic.mobiletest caused initialization of this class with the following trace: 
	at primitive_math$variadic_predicate_proxy.<clinit>(primitive_math.clj:27)

does order matter?

no. what is primitive_math?

I think the issue here may be that it's not in a package, just a bare class name

It's used by dtype-next

this is why you should probably never use a single segment namespace

😏 https://github.com/ztellman/primitive-math/blob/master/src/primitive_math.clj

wasn't me! 😛

I'll take this up with graalvm devs

you'll probably run into lots of classes from this library that you should explicitly list...

because it has no package

is it fixable without graalvm changes?

just by adding primitive_math$variadic_predicate_proxy and whatever else it complains about?

yes

but you'll likely need to enumerate quite few of them..

https://github.com/oracle/graal/discussions/3476#discussioncomment-922266

you could perhaps do this using the list of class files in the classes/primitive_math directory or so

I've added all the classes that were printed

compiling now

it will likely come up with a new class

if only they returned the error as edn 😛 (or even json).

{:unintentially-initialized ["primitive_math$variadic_predicate_proxy" ...]
 :error "Classes that should be initialized at run time got initialized during image building"}

escape the dollar?

the snippet was a wishful example of native image output

it seems like it's compiling

right

it compiled!

🎉

rather than using:

(defn initialize-at-build-time-list [& args]
  (println
   (->> (map ns-name (all-ns))
        (remove #(clojure.string/starts-with? % "clojure"))
        (map #(clojure.string/split (str %) #"\."))
        (keep butlast)
        (map #(clojure.string/join "." %))
        distinct
        (map munge)
        (cons "clojure")
        (clojure.string/join ","))))

@smith.adriane well, the point in the linked discussion is that not all classes should be initialized at build time, only the clojure ones

so if there are other Java classes in your jar then those should not be listed preferably

since native-image should figure it out by itself

oh, interesting

and the deadlock can happen because of initializing at build time for some of the Java stuff I think

but it remains to be seen if the above approach has really solved the issue. did you push the change? I could try it locally

Will do. one sec

I will also try this approach for https://github.com/borkdude/tools-deps-native-experiment which has a similar issue

ok, pushed the changes

It seems like it should be possible to generate the list of clojure classes since projects like depstar need to produce a list for AOT compilation

good thinking

Using the package for initialize at build time can have false positive since it's possible to have a java class that shares the same package as clojure code. I tried filtering all classnames in an uber jar by matching them with namespaces:

(defn list-resources [path]
  (let [jar (java.util.jar.JarFile. path)  
        entries (.entries jar)]
    (loop [result  []]
      (if (.hasMoreElements entries)
        (recur (conj result (.. entries nextElement getName)))
        result))))

(defn namespace->namespace-key [ns]
  (-> ns
      ns-name
      str
      (clojure.string/split #"\.")
      (->> (map munge))))

(defn class-path->namespace-key [fname]
  (-> fname
      (clojure.string/replace #"(\$[^.]+)?.class$" "")
      (clojure.string/split #"/"))
  )

(defn class-path->classname [fname]
  (-> (subs fname 0 (- (count fname)
                       (count ".class")))
      (clojure.string/replace #"/" ".")))

(defn clojure-classes-in-jar [jar-path]
  (let [resources (list-resources jar-path)
        ns-keys (into
                 #{}
                 (map namespace->namespace-key)
                 (all-ns))
        classes (into #{}
                      (comp
                       (filter #(clojure.string/ends-with? % ".class"))
                       (filter (fn [class-path]
                                 (contains? ns-keys (class-path->namespace-key class-path))))
                       (map class-path->classname))
                      resources)]
    classes))

(comment
  (spit "initialize-at-buildtime-classes.txt"
        (clojure.string/join ","
                             (clojure-classes-in-jar  "target/mobiletest-uber.jar")))

  ;; INITIALIZE_AT_BUILD_TIME=$(cat initialize-at-buildtime-classes.txt)
  ;;     --initialize-at-build-time=clojure,"$INITIALIZE_AT_BUILD_TIME" \
)

However it should be possible to modify the above code to include defrecord classes if including java classes that share packages with clojure namespaces isn't a big deal

> if including java classes that share packages with clojure namespaces isn't a big deal sometimes it is, in the case of httpkit for example

see its README

but that would also be the case with the namespace approach

I guess it should also be possible to list all classes created by defrecord and include them when filtering classes in an uberjar