Fork me on GitHub
Alex Miller (Clojure team)18:03:34

A new guide on how to use trusty compile to improve dev startup time

🎉 104
parrot 32
😍 12
👍 4
Alex Miller (Clojure team)18:03:16

Often the Clojure community survey receives many requests for faster startup time. There are however several distinct startup time use cases. One of those most frequently mentioned is development startup time (whether that's starting a repl that loads a user.clj or running a dev web server). It is not widely appreciated how easy it is to use Clojure's existing mechanisms to AOT compile your startup dependencies and use them while you develop, shaving 10s of seconds off of every start. Hopefully this guide will help. I am certain there is a lot that can be done in frequently used templates to leverage this idea and I would encourage any toolsmiths/template makers out there to do so! It requires no changes to Clojure itself, very few changes to how you work, and you can leverage it right now. There are of course, other use cases (lambda/serverless, production servers, etc). The core team continues to evaluate a range of options on those.


> If you are using a user.clj in dev, you need to force a reload because it has already been loaded automatically I'm not sure I get that. So it was loaded before I called compile. So what? If it wasn't changed, why would I want to reload it?

Alex Miller (Clojure team)19:03:12

compiling is a side effect of loading. If it's already been loaded, compilation won't happen


Gotcha, thanks! Huh. Would it then make sense to be able to just dump an already loaded ns into a class file? Assuming that it's theoretically possible.


Is this something you expect projects will setup themselves, or users?


Thank you Alex, this is something new I learned today. A sample project or more detailed guide where this technique is used on a large number of external dependencies could be even more helpful I believe especially for newer members of the community.


Just created this gist with the code that I use in production. Don't use in development yet because I didn't set up any invalidation, and forgetting to recompile stuff when you changed some dependency can lead to some non-trivial errors.

Alex Miller (Clojure team)20:03:36

@U2FRKM4TW that partially already exists - normally as you load namespaces, they are compiled and added to the dynamic classloader (at a function level). But, on disk things are a little different to facilitate file-oriented loading. I think you'd end up basically redoing the same work, doesn't seem like it would help much to have this.


@U064X3EF3 Hmm, I see. Could you please check out the comments in my gist linked above? Not sure if they have enough details, but maybe you have some ideas on how to deal with jars in general.

Alex Miller (Clojure team)20:03:11

@U09LZR36F what do you mean by "projects"?


@U064X3EF3 you're saying normally as you load namespaces, they are compiled and the guide article says compile ... compiles that namespace and all the namespaces it requires into *compile-path* - I wonder why 'normally' loading a namespace isn't doing that?


I must be missing something obvious I feel

Alex Miller (Clojure team)20:03:21

normally loading a clojure namespace will compile the namespace into a bunch of classes and load those into the dynamic classloader

Alex Miller (Clojure team)20:03:50

that all happens at runtime, dynamically

Alex Miller (Clojure team)20:03:03

the process in this article describes how to save and reuse that work


I see that, I was just wondering why the saving isn't happening automatically (provided that *compile-path* is on the classpath)? Thank you for the explanation and the example project!

Alex Miller (Clojure team)20:03:21

because it's not always needed - while you're working on your project, you probably don't actually want this at all

Alex Miller (Clojure team)20:03:31

for the namespaces in your own project


Huh. Interesting. I'm getting

Syntax error (IOException) compiling fn* at (user.clj:1:1).
No such file or directory
when trying to compile the example project.


24.75 -> 6.68 on my machine, that is a significant improvement!

❤️ 4

A minor nit: this line near the start has a smart quote in it, instead of a regular quote:

(compile ‘my.namespace)    ;; writes .class files to *compile-path*


Creating the dir did it. The error about fn* was especially unhelpful.


Oh, several of the code examples have those pesky smart quotes lower down too.

Alex Miller (Clojure team)20:03:24

I’ll fix those, wish I could uninvent those


@U064X3EF3 by project I mean something like tools.deps or the application I work on daily.


I'm not sure why but whenever I call compile on a ns, only that ns is compiled into classes directory - nses required by that ns are NOT compiled at all... I'm using leiningen.


I think lein does magic for that.


It seems like this approach would go well with some caching information. Library incoming...


Not sure about the classes issue above, but it it is git, this is normally solved by having a .gitkeep file inside it.

Alex Miller (Clojure team)20:03:02

maybe we should move this to #tools-deps b/c none of what you're saying makes sense to me


@U064X3EF3 sorry, not related to tools.deps at all. What I mean is, should something like core.cache be adding a dev/user.clj and compilation stuff? Or meander, oz, the lein-based project I work on daily, etc. Or should I, a developer, be setting this up for myself as a "personal workflow"?

Alex Miller (Clojure team)20:03:18

I think the answer for libs is generally, no

Alex Miller (Clojure team)20:03:04

user.clj is primarily used by people building applications (particularly common in web app templates, reloaded stuff, etc)

Alex Miller (Clojure team)20:03:40

if you are doing dev on a project, and it has a lot of deps, you might want to consider this. for those libs you mention, that's probably not the case.


I'm not suggesting libs would include this. Sorry, I'm not doing a good job. I mean for while you're developing those libraries.

Alex Miller (Clojure team)20:03:20

there is no reason to add a user.clj if you don't already have one


Okay. I think you've figured out what I mean now :).

Alex Miller (Clojure team)20:03:40

it's called out explicitly because it's special, and common for applications


Better phrasing would perhaps be, I work on 3 projects in a week. Should I update all 3 of them. Or should I update ~/.clojure/deps.edn.

Alex Miller (Clojure team)20:03:59

if the alias is identical, then sure, but I'm not sure if it would be


Wouldn't it be always extra paths classes? (Or maybe dominic_classes or something unique).

Alex Miller (Clojure team)21:03:03

not if you put it in a different path

Alex Miller (Clojure team)21:03:26

I can't answer these questions for you - they are dependent on your environment


I'm trying to figure out what the best current practice is. What I need to get projects to do, or my colleagues :).


35sec -> 11sec.

💥 4
Alex Miller (Clojure team)13:03:09

if you have files that you are actively changing all the time (b/c doing dev on them), aot compiling those namespaces is useless. but aot compiling your deps is still worthwhile

💯 4

sure; but wouldn’t it only do it if they had actually changed?


clojure is compiling anyway in memory when the repl evaluates; so why worry about putting those outputs on disk?

Alex Miller (Clojure team)13:03:38

I don't understand what you're asking


Sorry, you seemed to imply earlier that a user wouldn’t want their project namespaces to be compiled into classes/. I understand that because they’re changing frequently there’s no benefit to storing them. What I’m questioning though is whether there is a cost or problem if it is done. Otherwise it’s extra work for tooling to figure out which namespaces to pass to compile. Why do that work if doing so doesn’t cause people problems and is almost as quick?

Alex Miller (Clojure team)13:03:00

I don't think there's a problem, just saying it's not giving you a benefit


Ok cool. That’s all I wanted to clarify.


Interesting — I’d been meaning to look at what would be involved in doing something like this with some of our integrant systems which are essentially dynamically composed from configurations that are loaded through ig/load-namespaces. I’ll need to read this thoroughly to see if I can finally cobble something together. This is like we used to compile clojure code in the good old days! 😁


I think you can do a simple algorithm like: 1. ig/load-namespaces 2. analyze deps.edn for paths that are pulled in by the current project 3. collect all ns’s loaded from those path’s sources 4. (apply compile (set/difference (all-ns) *1))


Please do not have conversations in the #announcements channel.

👍 4

Or stick to a single thread under the original announcement.


Oops sorry sean — I thought this was a different channel!

👍 4

@U08E8UGF7: That seems a lot more fiddly than what I was thinking. Won’t it be awkward to know what paths/aliases your repl was running with? I was thinking you’d just implement compile-namespaces… probably something like:

(defn compile-namespaces [config keys]
  (doall (->> (ig/dependent-keys config keys)
              (mapcat #(conj (ancestors %) %))
              (mapcat ig/key->namespaces)


The whole point is to not compile the namespaces that you are working in on in the current project, since those will change frequently. You can’t determine where those integrant ns’es are coming from just from the config


anyway, let’s move this to #integrant

👍 4