Clojurians Log v2

Loop 10M iterations* 720ms 270ms

No JIT and SCI has good performance for a non-JIT interpreter apparently ;)

If I understand this correctly and reading the technical.md doc, the clojure core code is AOT compiled to avoid loading that from scratch but I guess loading clojure libs (which are generally not AOTed) would incur more cost as they are now compiled first to bytecode? should be slower than SCI then? at least to load the code?

assuming compiling clojure is slower than sci's eval?

the built-in clojure functions should be similar in bb and cream. the performance of loaded-from-source functions: haven't tested this yet, but the loop example shows better performance with bb so far. there is a JIT coming to Crema so things might change

yeah i would imagine the load time of various libs depending on the compiling complexity would hinder startup time. unless there's a way to cache that bytecode. like python does with pycache__. or shipping AOTed only clojure jars

would be interesting to measure the time it took for it to complete parsing and start eval for sci and cream given a source file for instance

well, go measure it

will give it a try

medley or camel-snake-kebab are probably good examples (pure clojure code, they both work in bb and cream)

how was this measured?

it's a bit of a bullshit comparison. these are my numbers with hyperfine locally:

$ hyperfine "./cream -e nil"
Benchmark 1: ./cream -e nil
  Time (mean ± σ):       3.8 ms ±   0.5 ms    [User: 1.3 ms, System: 1.6 ms]
  Range (min … max):     3.2 ms …   6.7 ms    416 runs

$ hyperfine "bb -e nil"
Benchmark 1: bb -e nil
  Time (mean ± σ):       8.3 ms ±   0.7 ms    [User: 2.8 ms, System: 3.6 ms]
  Range (min … max):     7.3 ms …  11.2 ms    225 runs

to measure loading time from source you could just do (time (require ...))

yep thats what i had in mind

seeing these numbers:

$ ./cream -Scp "$(clojure -Spath -Sdeps '{:deps {camel-snake-kebab/camel-snake-kebab {:mvn/version "0.4.3"}}}')" -M -e '(time (require (quote [camel-snake-kebab.core :as csk])))'
"Elapsed time: 125.03336 msecs"

$ bb -cp "$(clojure -Spath -Sdeps '{:deps {camel-snake-kebab/camel-snake-kebab {:mvn/version "0.4.3"}}}')" -e '(time (require (quote [camel-snake-kebab.core :as csk])))'
"Elapsed time: 26.231804 msecs"

$ ./cream -Scp "$(clojure -Spath -Sdeps '{:deps {dev.weavejester/medley {:mvn/version "1.9.0"}}}')" -M -e '(time (require (quote [medley.core :as mc])))'
Reflection warning, medley/core.cljc:519:25 - call to java.util.ArrayList ctor can't be resolved.
"Elapsed time: 120.566124 msecs"

$ bb -cp "$(clojure -Spath -Sdeps '{:deps {dev.weavejester/medley {:mvn/version "1.9.0"}}}')" -e '(time (require (quote [medley.core :as mc])))'
"Elapsed time: 32.24462 msecs"

nice.

we could add these to the README

can do a quick PR

https://github.com/borkdude/cream/pull/1

I think it would be interesting to also contrast the runtime performance of some examples, not only the load time. As just loading is something people will hardly do when executing a script.

Also: SCI has to assemble an AST that it will interpret, which takes time too, it's not just parsing. This all happens during require

i think clojure compile has to do the AST stuff too? I could imagine the bytecode emission could take most time?

should remove the runtime line i added?

removed that part of the line

I meant this part: > Pure clojure loading/parsing time from libraries would be slower than bb/SCI in cream as here it needs to first compile the clojure code into bytecodes and then run it. > Whereas bb interprets it directly through SCI without compilation.

LEt's just only state the timing, not the mechanism, it's unclear what makes it faster / slower to me. No assumptions without verifying. It could also be that the Clojure reader is slower, etc.

right

Also add a timing for actually executing some runtime code

maybe in a loop or so if it's below 50ms

and this code should be from the loaded lib?

what else?

if you're going to run a script, typically you load a library not just to make the script slower, but also use the library?

yep was just trying to think of a piece of code people would use from these libs

another interesting benchmark would be (aside from this PR) would be to compare clojure JVM vs cream on loading clojure.data.json and parsing 5MB of JSON. Back when clojure.data.json still ran with bb, this was very slow. Probably JVM Clojure still outperforms here (JIT).

would something like this be good enough?

$ bb -cp "$(clojure -Spath -Sdeps '{:deps {dev.weavejester/medley {:mvn/version "1.9.0"}}}')" -e '(time (require (quote [medley.core :as mc]))) (time (dotimes [i 100000] (mc/greatest 5 2 1 3 4)))'
"Elapsed time: 31.452928 msecs"
"Elapsed time: 347.639424 msecs"

$./cream -Scp "$(clojure -Spath -Sdeps '{:deps {dev.weavejester/medley {:mvn/version "1.9.0"}}}')" -M -e '(time (require (quote [medley.core :as mc]))) (time (dotimes [i 100000] (mc/greatest 5 2 1 3 4)))'

Reflection warning, medley/core.cljc:519:25 - call to java.util.ArrayList ctor can't be resolved.
"Elapsed time: 122.997416 msecs"
"Elapsed time: 785.789744 msecs"

interesting times too

yeah

I expect with ristretto, a JIT for Crema, cream would come out better. But this is still under development

mention that too probably

(https://github.com/oracle/graal/issues/11327#issuecomment-3914916673)

Very cool. Given that it’s operating at the bytecode layer, is Cream capable of loading libs that bb can’t?

amended the PR @borkdude

@henrik yes essentially its the JVM clojure experience, without the JIT (as of now)

yes, barring some Crema bugs at the moment with Class.forName and enums

How interesting, it’s kind of going full circle. I’ve been contemplating making https://github.com/multiplyco/quiescent bb-compatible, but since it relies on Java and whatnot, I’m not sure it would work without a dedicated bb code path. Just checked, it loads fine with cream OOB, but then explodes with a Fatal error: InvokeGeneric if I try to construct a task. Presumably more related to Crema than Cream itself.

./cream -Scp "$(clojure -Spath -Sdeps '{:deps {co.multiply/quiescent {:mvn/version "0.2.5"}}}')" \
  -M -e '(do (require (quote [co.multiply.quiescent :as q])) @(q/task 1))'

Give more details about how it crashes

The first few lines

Ok this is just a matter of preserving more classes

@henrik the issue is coming from this code: https://github.com/multiplyco/pathling/blob/v0.2.1/src/co/multiply/pathling/Transform.java#L36-L49 I'll try to narrow it down and paste an issue. it wasn't about preversing classes after all probably

Aha, surprising. I wonder what is throwing it off about the switch statement. The pattern matching feature itself has been stable since 21 IIRC, so it shouldn’t be unsupported as such.

Crema is just not complete yet I guess

Right, could just be basic alphaness

I will communicate it with the GraalVM team

@henrik I made some progress. It tuned out the above error could be fixed by including moar classes, but now I hit an issue with VarHandles Repro. Creating (.increment (WithVarHandle.)) with the following class in Crema crashes. I reported it

import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;

public class WithVarHandle {
    volatile int count;
    private static final VarHandle COUNT;
    static {
        try {
            COUNT = MethodHandles.lookup().findVarHandle(WithVarHandle.class, "count", int.class);
        } catch (Exception e) {
            throw new ExceptionInInitializerError(e);
        }
    }
    public WithVarHandle() { this.count = 0; }
    public int increment() { return (int) COUNT.getAndAdd(this, 1); }
    public int getCount() { return count; }
}

Wow, thanks!

They’re an important part of my (pathological, according to my psychologist) quest to avoid object allocations.

I added support for running .java files now too: https://github.com/borkdude/cream?tab=readme-ov-file#running-java-files So you can use it as a faster jbang :)