Clojurians Log v2
clojure-dev
Is there any way of obtaining all the class files produced by the AOT compilation of a namespace? My reading of the code says no, but I’d love to be proved wrong.
That would be useful for graalvm purposes as well
sounds like you want to separate the classes produced by aot namespace A from the class files produced by namespace B, which A depends on
For my use case (ensuring that incremental compilation in IntelliJ works) I’d want both, but I can imagine that separating them would also be useful (e.g. I only want to AOT my classes, and I’m happy to have the rest run-time compiled).
clojure basically behaves like some kind of whole program optimizing compiler, despite not doing any of things those do
so like taking apart the output of an optimizing compiler and linking it with something compiled separately is tricky, so is doing that with aot compiled clojure
Right, I’m assuming that if you compile several namespaces, that the dependent classes are re-written each time. I haven’t checked that, though.
the name of classes produced by the compiler are not deterministic
no, they're not (answering cfleming's question)
classes that exist are not recompiled because they are not loaded because they've already been loaded
Ah, that makes sense.
going back to original question, do you mean "including transitively compiled classes"? or just from that namespace
I really wish the names were deterministic, it would help debugging as well.
In my case, including transitive classes. When Clojure is AOT compiled in Cursive, I compile a namespace at a time (due to how the IntelliJ compiler framework works). In order for incremental compilation not to get messed up (not just for Clojure but for other compilers that might be working in the same project) I should be able to produce a mapping from source files to output files.
Thinking about the transitive case, that’s tricky because which class files are produced depends on which namespaces have been previously compiled (the not loaded thing).
it also depends on loaded
Currently I just map the source file to the ns initialisation class, which works for detecting which namespaces need to be compiled, but probably leaves lots of crud around in the output directories.
because if a transitive namespace is already loaded, it won't be loaded again when compiling, so no classfiles will be written to disk
why is this different than the Java model it's presumably written for?
Java source files produce N class files and depend on M other classes (which may overlap)
Because when you compile a Java file, you can deterministically figure out which class files will be produced.
Right, but compiling one Java file doesn’t produce class files from other classes.
so you need this a priori?
Well, Cursive has always done this and no-one has complained, so it’s probably not an urgent problem. I suspect very few people use the AOT compilation feature anyway. But as borkdude mentioned, it would be a useful feature for other cases too, like graalvm.
I understand that it’s a hard problem due to how the Clojure compiler has always worked, though.
like there are a set of name patterns you could look for (foo, foo__init, foo$... etc) but you're right that there is no way to know the actual set of classes until you compile.
For a graalvm plugin I wrote I needed to know every class produced by clojure. I worked around this by collecting all the namespace names and using that as package wildcards, but it's better if you could reliably get the collection of class names.
I also ran into the non-deterministic names
And doing that, you wouldn’t get the transitive ones either, right?
The thing is, for a particular namespace compilation, it’s not even deterministic which namespace’s files will be created, since the dependency namespaces might have been previously loaded.
But for my AOT case, I probably want to figure out the transitive closure of dependent namespaces for each namespace I compile, and then return all those namespaces’ classes as potentially resulting from the compilation of the original source file.
Since if that namespace is compiled cold with an empty output directory, all those files will be produced.
well, the dependency is really the reverse direction and ideally you'd compile those in the other direction
that's the effect you're getting but via side effects
Right.
Ok, I’ll take a look and see if all that is worth it. I’m not keen on scanning the output directory, but it looks like that’s really the only option.
In my case, the set of files to compile is passed to me by the IntelliJ framework, and doesn’t include the dependencies from libraries. So I’ll always be compiling top-down, I can’t compile bottom up.
At least, if that is possible I haven’t figured out how to do it.
unless you analyze those files yourself maybe and then figure out the bottom dependencies?
Yes, I could do that, and in order to correctly create the output set I’ll have to do that (i.e. pass the tree of namespace dependencies to the compiler).
the tools build compile clj task does something like that
the bug that resulted in that being added might be interesting reading as well
oh, guess the discussion is all in slack and not in jira https://clojure.atlassian.net/browse/TBUILD-7
basically compiling each namespace in some random order would break things, but in dependency order wouldn't
Interesting, thanks, I’ll see if I can dig out the slack conversation.
Yeah, there’s lots of interesting things in #tools-build if you search for “topo”
I think in #tools-deps too