Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2020-03-01

Channels

#tools-deps
<
2020-03-01
>
sam22:03:57

I wrote a script to aot-compile classes in my project, which I have configured to run automatically when necessary via aliases. e.g. 

:test      {:extra-paths ["test" "test-resources"]
                       :extra-deps {com.cognitect/test-runner {:git/url ""
                                                               :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}}
                       :main-opts ["-i" "aot.clj"
                                   "-m" "cognitect.test-runner"]}
I tried adding logging to my aot script using clojure.tools.logging, and found that tests started failing with java.lang.NoClassDefFoundError: clojure/tools/logging/impl/LoggerFactory I don’t totally understand what’s going on, but I think it has something to do with the fact that the clojure compile function transitively compiles only namespaces that haven’t already been loaded. So, because I have already loaded clojure.tools.logging before calling compile, it does not get transitively compiled even though the namespaces I’m compiling depend on it. Still, I wasn’t sure that this situation should cause a NoClassDefFoundError, since the clojure source code should still be available on my classpath, but I think it might be due to this issue: https://clojure.atlassian.net/browse/CLJ-1544. If I understand correctly, I think it’s important in my case for clojure.tools.logging to be compiled since the calling code is compiled. My workaround was to add an explicit (compile 'clojure.tools.logging) to the script. So my questions are: 1. Does my understanding of the problem make sense? 2. Is there an issue here with compile that should be addressed? Assuming my understanding of all this is correct, I wish compile had options about how it should handle dependency namespaces, or at least that the documentation specified this behavior (if it does, I couldn’t find it).

seancorfield23:03:20

@samwagg0583 AOT has a number of serious "gotchas" and it's best to avoid it unless you absolutely need it (it's "safe" as the very last step before you build an uberjar for deployment as a complete application, but most other uses can trip you up).

Alex Miller (Clojure team)23:03:22

I don’t think this is CLJ-1544

Alex Miller (Clojure team)23:03:40

Compile is transitive but it can’t necessarily “see” everything and you may need to explicitly compile things sometimes. All that is normal.

seancorfield23:03:50

When code is (partially) AOT'd and then it attempts to load in (and compile on-demand) more code, you can get mismatches between the classes that the class loaders can see. Is that a fair description @alexmiller?

Alex Miller (Clojure team)23:03:57

It’s more specific than that

Alex Miller (Clojure team)23:03:51

Your issue with having it already loaded is something to consider and you may need to change your compilation process to account for that

seancorfield23:03:17

@samwagg0583 What is driving your need to AOT code for testing here?

sam23:03:14

We are using a java library for distributed data pipelines called Apache Beam and running in Google Cloud Dataflow. Running a pipeline involves running code on a local JVM that uploads serialized java objects to remote JVMs. De-serialization on the other end requires class files, hence the AOT.

sam23:03:31

My workaround of explicitly compiling classes that are already loaded seems to work okay. I just wanted to better understand the issue.