Fork me on GitHub
#announcements
<
2020-03-12
>
alexmiller18:03:34

A new guide on how to use trusty compile to improve dev startup time https://clojure.org/guides/dev_startup_time

alexmiller18:03:34

A new guide on how to use trusty compile to improve dev startup time https://clojure.org/guides/dev_startup_time

alexmiller18:03:16

Often the Clojure community survey receives many requests for faster startup time. There are however several distinct startup time use cases. One of those most frequently mentioned is development startup time (whether that's starting a repl that loads a user.clj or running a dev web server). It is not widely appreciated how easy it is to use Clojure's existing mechanisms to AOT compile your startup dependencies and use them while you develop, shaving 10s of seconds off of every start. Hopefully this guide will help. I am certain there is a lot that can be done in frequently used templates to leverage this idea and I would encourage any toolsmiths/template makers out there to do so! It requires no changes to Clojure itself, very few changes to how you work, and you can leverage it right now. There are of course, other use cases (lambda/serverless, production servers, etc). The core team continues to evaluate a range of options on those.

p-himik19:03:08

> If you are using a user.clj in dev, you need to force a reload because it has already been loaded automatically I'm not sure I get that. So it was loaded before I called compile. So what? If it wasn't changed, why would I want to reload it?

alexmiller19:03:12

compiling is a side effect of loading. If it's already been loaded, compilation won't happen

p-himik19:03:27

Gotcha, thanks! Huh. Would it then make sense to be able to just dump an already loaded ns into a class file? Assuming that it's theoretically possible.

dominicm19:03:30

Is this something you expect projects will setup themselves, or users?

imre19:03:42

Thank you Alex, this is something new I learned today. A sample project or more detailed guide where this technique is used on a large number of external dependencies could be even more helpful I believe especially for newer members of the community.

p-himik20:03:35

Just created this gist with the code that I use in production. Don't use in development yet because I didn't set up any invalidation, and forgetting to recompile stuff when you changed some dependency can lead to some non-trivial errors. https://gist.github.com/p-himik/fcf638b05d70d908c74d98e25fd9e935

alexmiller20:03:36

@U2FRKM4TW that partially already exists - normally as you load namespaces, they are compiled and added to the dynamic classloader (at a function level). But, on disk things are a little different to facilitate file-oriented loading. I think you'd end up basically redoing the same work, doesn't seem like it would help much to have this.

p-himik20:03:50

@U064X3EF3 Hmm, I see. Could you please check out the comments in my gist linked above? Not sure if they have enough details, but maybe you have some ideas on how to deal with jars in general.

alexmiller20:03:11

@U09LZR36F what do you mean by "projects"?

imre20:03:25

@U064X3EF3 you're saying normally as you load namespaces, they are compiled and the guide article says compile ... compiles that namespace and all the namespaces it requires into *compile-path* - I wonder why 'normally' loading a namespace isn't doing that?

imre20:03:35

I must be missing something obvious I feel

alexmiller20:03:21

normally loading a clojure namespace will compile the namespace into a bunch of classes and load those into the dynamic classloader

alexmiller20:03:50

that all happens at runtime, dynamically

alexmiller20:03:03

the process in this article describes how to save and reuse that work

imre20:03:32

I see that, I was just wondering why the saving isn't happening automatically (provided that *compile-path* is on the classpath)? Thank you for the explanation and the example project!

alexmiller20:03:21

because it's not always needed - while you're working on your project, you probably don't actually want this at all

alexmiller20:03:31

for the namespaces in your own project

p-himik20:03:07

Huh. Interesting. I'm getting

Syntax error (IOException) compiling fn* at (user.clj:1:1).
No such file or directory
when trying to compile the example project.

alexmiller20:03:03

mkdir classes

imre20:03:37

24.75 -> 6.68 on my machine, that is a significant improvement!

seancorfield20:03:55

A minor nit: this line near the start has a smart quote in it, instead of a regular quote:

(compile ‘my.namespace)    ;; writes .class files to *compile-path*

p-himik20:03:12

Creating the dir did it. The error about fn* was especially unhelpful.

seancorfield20:03:23

Oh, several of the code examples have those pesky smart quotes lower down too.

alexmiller20:03:24

I’ll fix those, wish I could uninvent those

dominicm20:03:22

@U064X3EF3 by project I mean something like tools.deps or the application I work on daily.

jumar20:03:00

I'm not sure why but whenever I call compile on a ns, only that ns is compiled into classes directory - nses required by that ns are NOT compiled at all... I'm using leiningen.

dominicm20:03:30

I think lein does magic for that.

dominicm20:03:17

It seems like this approach would go well with some caching information. Library incoming...

dominicm20:03:24

Not sure about the classes issue above, but it it is git, this is normally solved by having a .gitkeep file inside it.

alexmiller20:03:02

maybe we should move this to #tools-deps b/c none of what you're saying makes sense to me

dominicm20:03:34

@U064X3EF3 sorry, not related to tools.deps at all. What I mean is, should something like core.cache be adding a dev/user.clj and compilation stuff? Or meander, oz, the lein-based project I work on daily, etc. Or should I, a developer, be setting this up for myself as a "personal workflow"?

alexmiller20:03:18

I think the answer for libs is generally, no

alexmiller20:03:04

user.clj is primarily used by people building applications (particularly common in web app templates, reloaded stuff, etc)

alexmiller20:03:40

if you are doing dev on a project, and it has a lot of deps, you might want to consider this. for those libs you mention, that's probably not the case.

dominicm20:03:49

I'm not suggesting libs would include this. Sorry, I'm not doing a good job. I mean for while you're developing those libraries.

alexmiller20:03:20

there is no reason to add a user.clj if you don't already have one

dominicm20:03:20

Okay. I think you've figured out what I mean now :).

alexmiller20:03:40

it's called out explicitly because it's special, and common for applications

dominicm20:03:44

Better phrasing would perhaps be, I work on 3 projects in a week. Should I update all 3 of them. Or should I update ~/.clojure/deps.edn.

alexmiller20:03:59

if the alias is identical, then sure, but I'm not sure if it would be

dominicm21:03:24

Wouldn't it be always extra paths classes? (Or maybe dominic_classes or something unique).

alexmiller21:03:03

not if you put it in a different path

alexmiller21:03:26

I can't answer these questions for you - they are dependent on your environment

dominicm21:03:41

I'm trying to figure out what the best current practice is. What I need to get projects to do, or my colleagues :).

ikitommi06:03:47

35sec -> 11sec.

alexmiller13:03:09

if you have files that you are actively changing all the time (b/c doing dev on them), aot compiling those namespaces is useless. but aot compiling your deps is still worthwhile

rickmoynihan13:03:12

sure; but wouldn’t it only do it if they had actually changed?

rickmoynihan13:03:11

clojure is compiling anyway in memory when the repl evaluates; so why worry about putting those outputs on disk?

alexmiller13:03:38

I don't understand what you're asking

rickmoynihan13:03:00

Sorry, you seemed to imply earlier that a user wouldn’t want their project namespaces to be compiled into classes/. I understand that because they’re changing frequently there’s no benefit to storing them. What I’m questioning though is whether there is a cost or problem if it is done. Otherwise it’s extra work for tooling to figure out which namespaces to pass to compile. Why do that work if doing so doesn’t cause people problems and is almost as quick?

alexmiller13:03:00

I don't think there's a problem, just saying it's not giving you a benefit

rickmoynihan13:03:19

Ok cool. That’s all I wanted to clarify.

rickmoynihan20:03:35

Interesting — I’d been meaning to look at what would be involved in doing something like this with some of our integrant systems which are essentially dynamically composed from configurations that are loaded through ig/load-namespaces. I’ll need to read this thoroughly to see if I can finally cobble something together. This is like we used to compile clojure code in the good old days! 😁

kszabo20:03:58

I think you can do a simple algorithm like: 1. ig/load-namespaces 2. analyze deps.edn for paths that are pulled in by the current project 3. collect all ns’s loaded from those path’s sources 4. (apply compile (set/difference (all-ns) *1))

seancorfield20:03:15

Please do not have conversations in the #announcements channel.

seancorfield20:03:35

Or stick to a single thread under the original announcement.

rickmoynihan20:03:47

Oops sorry sean — I thought this was a different channel!

rickmoynihan09:03:48

@U08E8UGF7: That seems a lot more fiddly than what I was thinking. Won’t it be awkward to know what paths/aliases your repl was running with? I was thinking you’d just implement compile-namespaces… probably something like:

(defn compile-namespaces [config keys]
  (doall (->> (ig/dependent-keys config keys)
              (mapcat #(conj (ancestors %) %))
              (mapcat ig/key->namespaces)
              (distinct)
              (compile))))

kszabo09:03:30

The whole point is to not compile the namespaces that you are working in on in the current project, since those will change frequently. You can’t determine where those integrant ns’es are coming from just from the config

kszabo09:03:00

anyway, let’s move this to #integrant