Fork me on GitHub
#clojure-dev
<
2018-09-21
>
dominicm17:09:28

I just ran into the fact that compile doesn't compile anything that's already loaded. I can't seem to see anywhere that specifies this behaviour. Is this a long standing bug with no good / intended fix? Is this documented somewhere and I can add that place to my check list when understanding things.

alexmiller17:09:22

compilation happens as a side effect of load so that makes conceptually makes sense to me

alexmiller17:09:02

https://clojure.org/reference/compilation is probably the canonical reference page

alexmiller17:09:22

I wouldn’t say it’s a bug and it seems like it it worked otherwise, compilation would result in a lot of recompilation when compiling many namespaces

dominicm17:09:26

It's a problem because the root namespace doesn't look for the .clj too, so it actually fails when run.

alexmiller17:09:04

don’t understand

dominicm17:09:07

In my case I did an AOT of user (root ns), edge.system was already loaded though, so didn't AOT. When I tried to use user as a main from that directory later, it failed because it couldn't find edge/system$new_system.class

dominicm17:09:33

Even though edge/system.clj was present. Message didn't indicate that it tried both.

ghadi17:09:43

To clarify, you AOTed 'user, which was already loaded by the time you asked for 'user to get compiled

👍 1
dominicm17:09:32

I'm very bad at articulating things like this, I apologize.

alexmiller17:09:34

given that user is special-cased into the runtime, this is a particularly weird case

alexmiller17:09:45

so I’d suggest not doing that :)

dominicm17:09:13

I agree. I think it would happen in other cases too. I'm certain I saw it a while ago in another project not using user.

ghadi17:09:27

It actually doesn't matter which ns you AOTed - any thing required by user would be affected

alexmiller17:09:58

exactly - user is designed NOT to be AOT’ed (as the runtime explicitly loads user.clj)

dominicm18:09:42

But if I have a user.clj in my classpath that requires edge.main, that would cause this bug too.

dominicm18:09:56

Or if I run compile from the repl.

alexmiller18:09:26

so, don’t load stuff in user.clj?

dominicm18:09:38

That's a fine answer, if it's the intended behaviour. It does cause minor issues trying to write a build system that doesn't have to spin up a whole extra jvm/pod just for aot.

dominicm18:09:39

There are workarounds though, so nothing pressing. But if it's something that could be enhanced, then I'm happy to open a jira, or vote for one. If there's a reason that it should stay this way then that's knowledge worth having too.

alexmiller18:09:33

I said the reasons above

alexmiller18:09:55

if compile recompiled loaded stuff then every compile would recompile everything

dominicm19:09:00

Why is that a problem? I guess that's what I'm missing?

alexmiller19:09:07

Because most people don’t their compile time to be n!

alexmiller19:09:07

I guess more n^2 or something

dominicm19:09:26

I suppose the caching seems wrong to me. I would expect caching to be based on the class file, not what is loaded in memory, and possibly not on disk.

dominicm19:09:49

What is the expectation for use here? Where this form of caching is advantageous.

alexmiller19:09:32

it’s just leveraging the existing load system which transitively loads namespaces. if you encounter a new (unloaded) namespace, compile it

alexmiller19:09:31

you’re N steps down the decision chain here, asking “why X”. it’s difficult to walk that chain backwards, recovering all the design decisions from 10 years ago.

dominicm20:09:14

Leveraging the load system I understand. It's a fairly convenient way to implement this. Sorry, I think I have misunderstood. I thought you were implying that the expected common case for compile involved a scenario where skipping loaded namespaces was useful. I was trying to figure out what the expected workflow is exactly. I understand that performance is a feature, but I wouldn't expect it at this kind of detriment to compile. With limited experience with the larger picture for this feature, this limitation seems arbitrary, verging on accident of history. Which is okay too, I am not interested in the why of how it happened so much as why it should stay the way it is.

andy.fingerhut21:09:08

I don't have experience in dealing with the detailed behavior of compile here, so out of my depth, but in general a reason for things to stay the way they are is to avoid creating possible bugs in build and/or deployment flows that people are using today.

alexmiller21:09:37

“I thought you were implying that the expected common case for compile involved a scenario where skipping loaded namespaces was useful.” - yes I was saying that

alexmiller21:09:11

if you are in a project and you want to compile everything in the project, you need to invoke compile for every namespace in the project

alexmiller21:09:50

you don’t want every one of those calls to recompile every namespace

alexmiller21:09:03

you want to compile every namespace once

alexmiller21:09:26

by compiling on load and leveraging the set of loaded namespaces as a tracking mechanism, you achieve that