Fork me on GitHub
#clojure-dev
<
2015-11-13
>
seancorfield05:11:31

Finally got a chance to run our code base with direct linking enabled for everything tonight. Breaks our test suite but only in a minor way: we have Expectations and use side-effects which does a redefinition, which of course requires ^:dynamic.

seancorfield05:11:53

After I'd annotated a handful of functions with ^:dynamic everything passed.

seancorfield06:11:29

Overall there seems to be a small performance improvement but I'll need to do more benchmarking to see how it will work in practice.

seancorfield06:11:28

As far as the tests are concerned, it'll be easy enough to have direct linking enable for non-dev/CI runs but it highlighted that some of our (early) tests rely on with-redefs where there don't need to (we should refactor to improve composability instead).

bronsa20:11:11

@alexmiller: do you think it would be possible to release 1.8 (when the time comes) both direct linked and "normal"?

bronsa20:11:02

just the way for example core.typed or tools.reader are released both JIT and AOT

bronsa20:11:25

(using different :scopes? I don't really understand how that works simple_smile)

Lambda/Sierra20:11:06

@bronsa: There's already the -slim JAR release, which contains just .clj files and compiled Java.

Lambda/Sierra20:11:24

That might work as a non-direct-linked version.

bronsa20:11:40

@stuartsierra: I'm aware of that, but AOT non-direct is still faster to load than JIT

bronsa20:11:30

so yeah, -slim might work if somebody wants to use 1.8 but can't use DL

bronsa20:11:44

but a 1.8 AOT non DL would be even better simple_smile

bronsa20:11:05

I don't think it makes sense to distribute clojure DL by default

bronsa20:11:19

ideally I think we should distribute a "dev" clj with DL and inlining off and a "prod" clj with DL and inlining on

bronsa20:11:06

yeah that's exactly what I'm thinking of simple_smile

bronsa20:11:00

I feel like getting build profiles is so much more necessary with DL

bronsa20:11:19

I want with-redefs to work on my repl

bronsa20:11:43

both :inline and direct linking are stuff that's really good for production code but doesn't play well in a dev env

Lambda/Sierra20:11:23

There's a downside too: people may see something work in a REPL and be surprised when it doesn't work in production. AOT-compilation already creates headaches like that.

Alex Miller (Clojure team)21:11:24

the downside of the build profile stuff (especially if you actually want to run tests for both) is that it makes the build much more complicated, esp if you want to release all the profiles as the same "build"

Alex Miller (Clojure team)21:11:59

I have a lot of work on it stashed away somewhere. I actually ended up removing all the legacy ant stuff in the process.

Alex Miller (Clojure team)21:11:58

I spent a lot of time in 2014 thinking about this b/c technically you could have the same question for all of your lib dependencies so you potentially get an explosion of combinations. is that useful? prob not. maybe it's not even a real problem since most people don't aot libs.

Alex Miller (Clojure team)21:11:54

we considered also removing all the non-essential metadata (docstring, etc) for the prod build as that stuff ends up in the constant pool and does have some effect on loading, size, etc.

Alex Miller (Clojure team)21:11:43

this idea seems at odds with where we are in the release cycle though

ghadi22:11:37

release cycle is whatever you decide it to be

seancorfield23:11:00

For consistency of testing, I would much prefer using "production-like" code for dev and testing too.

seancorfield23:11:34

I got our dev build/test working with full DL but at the expense of explicitly annotating a few functions with ^:dynamic purely because we mock them in tests.

seancorfield23:11:06

My goal now is to change the code and how we test it so we don’t need to do that, so we can use DL for dev without workarounds.

ghadi23:11:20

linking vars with invokedynamic should give production performance with no loss of dynamicity

ghadi23:11:38

big air quotes on "production performance".

seancorfield23:11:52

And that requires a compiler change and being Java 7+ only simple_smile

ghadi23:11:50

anything with bytecode requires a compiler change

ghadi23:11:14

i've had invokedynamic running on vars for a while, haven't ported it part 1.6

seancorfield23:11:21

Maybe now we have DL, Clojure/core will be persuaded to drop Java 6 and implement invokedynamic? simple_smile

ghadi23:11:51

the blocker for invokedynamic at this point is android and our tens of android users

ghadi23:11:56

indy is more performant after 8u20 -- for extreme users like Nashorn/jruby. for clojure usages are relatively minimal

seancorfield23:11:36

Do we know what % of Clojure users are on which versions of Java these days?

ghadi23:11:06

no offense to clojure-android users, I just don't feel like it merits holding the JVM back

Alex Miller (Clojure team)23:11:18

well, Colin has also raised the issue of being unable to move Cursive off jdk 1.6

ghadi23:11:28

indy could be certainly handled with a flag. whether it interferes with ProGuard is tbd.

Alex Miller (Clojure team)23:11:36

I have a State of Clojure survey prepped and hope to put it out after the conj

Alex Miller (Clojure team)23:11:43

had hoped to do it this week, but too busy

ghadi23:11:03

Cursive could be handled with a flag

ghadi23:11:45

they're running a fork anyways

ghadi23:11:01

cursive/intellij can run and debug code on Java 8, including lambdas (which are implemented with invokedynamic). what's the problem?

tcrayford23:11:14

@ghadi @alexmiller I think IntelliJ 15 or whatever the latest is fixes cursive's reliance on 1.6 JDK

Alex Miller (Clojure team)23:11:17

Ask @cfleming for details- he's been the biggest advocate for keeping 1.6

tcrayford23:11:06

The story is hella funny: oracle never have and won't ship a OSX JVM with good font rendering in it's GUI toolkits. So instead, every IntelliJ/eclipse/etc user ends up using apple's very old fork of 1.6 JVM to get font rendering that's reasonable. Jetbrains even recommend using it (even though it's old and has known security flaws). IntelliJ 15 iirc has a Jetbrains fork of 1.8 JDK that fixes font rendering on OSX

tcrayford23:11:39

But obviously that doesn't help users on old IntelliJ versions. Cursive needs JVM 1.6 support because it has to run in the JVM that the IDE is in, not the JVM your application code is in

ghadi23:11:37

That is hella funny... Invokedynamic could be opt in (like direct linking)