This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-09-12
Channels
- # announcements (2)
- # bangalore-clj (4)
- # beginners (97)
- # calva (57)
- # cider (24)
- # clj-kondo (1)
- # cljdoc (4)
- # cljs-dev (8)
- # cljsrn (6)
- # clojure (27)
- # clojure-berlin (7)
- # clojure-dev (95)
- # clojure-europe (16)
- # clojure-italy (15)
- # clojure-losangeles (9)
- # clojure-nl (8)
- # clojure-spec (10)
- # clojure-uk (29)
- # clojurescript (25)
- # clojutre (2)
- # clr (6)
- # cursive (33)
- # datomic (20)
- # dirac (1)
- # duct (9)
- # fulcro (2)
- # graalvm (18)
- # jobs (5)
- # jobs-discuss (19)
- # nrepl (4)
- # nyc (1)
- # off-topic (18)
- # pathom (15)
- # re-frame (22)
- # reagent (4)
- # reitit (1)
- # rewrite-clj (9)
- # spacemacs (2)
- # xtdb (12)
Hi. Would like to revisit the client-side compilation cache for libraries (https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6). I think this is really important for the developer ux as the batteries-included web stacks have a load time of 10+sec just to produce a hello world server. If the dependencies haven’t changed (mostly dont between repl startups), the compiler would use client-side AOTed classed. Is there any work related to this ongoing? Is this something that someone from the non-core team could try to solve as I guess you Alex & Rich are busy doing all other things too?
Here’s a example what happens when adding 2 useful utility libs to an utility lib:
:; just the code
(time (require '[malli.core]))
"Elapsed time: 273.110534 msecs"
;; with borkdude/sci
(time (require '[malli.core]))
"Elapsed time: 1303.306083 msecs"
;; with borkdude/sci + borkdude/edamame
(time (require '[malli.core]))
"Elapsed time: 1913.595456 msecs"
I think this is in Alex's list, I asked about this a while ago in this channel. In particular I was frustrated about the load time of core async.
I have something that cuts core.async load time by 5 when AOTed @dominicm , compatibly
What about without AOT? My problem is that it takes 10s to require a library if it requires core async, making the library seem "heavy". It's largely aesthetic.
For me, the caching is more interesting. I actually don't care too much about production start-up time.
understood, but caching will get no faster than AOT, because caching relies upon the AOT artifacts
For sure, admittedly I'd assumed that core async was reasonably fast when AOTd, but that's probably not the case if you're looking into it.
Yeah, it really matters there. Although I really like the datomic ions model and I'll probably do the thin lambda model next time.
I can't make sense of the timings above without researching what is being loaded. Do you know @ikitommi ?
@ghadi In my example, the actual libraries don’t matter. The library source gets recompiled every time a repl starts, even if the dependencies haven’t been changed. Local cache would be sweet. Here’s load time of Schema on my macbook:
➜ ~ clj -Sdeps '{:deps {prismatic/schema {:mvn/version "1.1.12"}}}'
Clojure 1.10.0
user=> (time (require '[schema.core :as s]))
"Elapsed time: 1117.321478 msecs"
nil
but in my case, both sci & edamame use a (different) inlined version of tools.reader, discussed with @borkdude about that, will make a PR where tools.reader is used as dependency. Should make the code load faster.
in the case of core.async: no. It loads tools.analyzer, tools.reader, and the go compiler, when it doesn't need it
so that once they are locally compiled, the locally compiled (AOT’ed) version would be used if the deps don’t change? that would be totally awesome
you could do AOT stashing with changing deps, as long as the cache key is sufficiently smart @ikitommi
many projects with unnecessary dependencies (cultural issue) dependency granularity can be large
I suppose if the user doesn't load an aot version in their code, the aot version will be loaded as priority?
@dominicm I’ve certainly seen issues around ns order of reloading when mixing AOT and JIT stuff together over the years
I’m not sure what context you are aiming at - is this for dev-time? I’d think that could potentially get annoying due to these sort of concerns - but I guess I’m not the definitive source of wisdom here.
I’ve just had to track some really tricky classloader problems in the past around JIT compile reloads and AOT loaded classes
Also, keep in mind that an AOT’ed lib, by default, does not know it’s own “boundaries”
it’ll AOT all it’s own deps at the same time. build tools, like lein
have a feature that strips out the non-lib-project’s class files to avoid problems with this
so the issue here ends up being when you AOT 2 of your own libs A and B, and they both use C
both will AOT C, and if C is not the same between the two, you will have classfiles for both on classpath
good. distributing source seems to have other advantages as well (source-compatibility is perhaps stronger than “binary” across clj versions, end-user gets to choose how to compile their “whole app”, with things like direct-linking, etc)
but anyways, @ghadi knows far more than I do on this topic, so I shouldn’t be chiming in I think hah
discussed in this old issue https://clojure.atlassian.net/browse/CLJ-322
but some build tools (`lein` is really the one I know does this), will remove the dependency classfiles after the AOT is complete - which is mentioned in the closing remarks here https://clojure.atlassian.net/browse/CLJ-322?focusedCommentId=36994&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-36994
not if you place the source for JIT those on the classpath during the actual runtime ?
this is how it ends up working if you do use lein
to AOT a lib project (not something I like to do still)
The AOT’ed classes should be able to initiate their dependency JIT again via their load
forms they initialize with
there are cases where AOT is essential of course, for direct interop things in some cases
> and the project’s dependencies override the libs with an incompatible version I think you mean?
lein's approach isn't something where say rich sat down and said "oh this will work well with how aot compilation works" lein's approach is phil got frustrated with how transitive aot compilation works, and just decided to kludge deleting some stuff, and that'll be fine right?
I mean, consuming via AOT’ed stuff puts more strictness on what source you can reliably JIT - in particular, around interop details - such as hints
either way, I won’t drag it on. I’m not a fan of AOT’ing libs - so not really taking a stance in favor of it.
If any sort of AOT cache is introduced, there had better be a way to disable it -- or to not have it as default and need to opt into it. Given all we know about how problematic it can be at the file/lib level, I dread to think what sort of weird problems beginners are going to run into if CLI/`deps.edn` forces this on them 😞
(I'm no fan of AOT as it currently works and avoid it at all costs)
oh, hah, here I am commenting on clj-322 about why boot's shift isn't a complete solution, and lein's removing of transitive class files was turned off by default at some point(I don't know the current status of that feature) with a link to the lein issue showing what it broke and why it was turned off
it doesn't look like it does it by default, even though the sample.project.clj shows setting it to true, but it is hard to tell
https://github.com/technomancy/leiningen/blob/master/src/leiningen/compile.clj#L105-L111 it only deletes if that key is set in the project, and a search in the repo doesn't show that key being set anywhere by default (sample.project.clj does show that key set with the opposite of its default value)
@hiredman recently I saw people at work AOT’ing libs and i was concerned - but then noticed it seemed to automatically be doing this clean
easy enough to check though - you could be right. I just am going from an experience in one of the recent’ish lein
versions. also, I guess this is a topic for lein
at that point.
libs people use are compiled when they're loaded, a cache would be about skipping the compilation
![aw_yeah](https://emoji.slack-edge.com/T03RZGPFR/aw_yeah/e2b893569649f890.gif)
Are there perhaps libs that are just so danged dynamic in the compiled code they generate, based upon run-time factors when they are loaded, that they would be too difficult to cache? I don't have an example in hand, but there must be some libs that change what they def/defn based upon some JVM property string or something.
or custom environment variables
yeah, a cache on the tool side, where the tool can observe the inputs and see if they change vs. aot compiling and publishing the artifacts is the most likely way to have it actually work
I'd be OK with the tooling compiling and caching non-local libs I depend on, but I wouldn't want it compiling and caching either :local/root
or :paths
dependencies.