Fork me on GitHub
#joker
<
2020-02-19
>
jcburley02:02:09

I've created a "proper" PR for my startup-time-optimization work -- please try it out and let me know how it compares, performance-wise (and of course whether it works!), to "official" Joker: https://github.com/candid82/joker/pull/358

jcburley18:02:31

This was merged by @roman.bataev last night (thanks!!). Please try it out, by pulling down the latest master, building, and testing it. (It’s working great for me so far on my email server, where it’s noticeably faster starting up.)

jcburley18:02:18

I should mention (since I got bit by this myself just now, during a demo) that, to install the fast-startup version of Joker, after running run.sh or go generate ./..., please do:

$ go install -tags fast_init
Or just copy joker.fast to wherever you want it, naming/linking it "`joker`".

borkdude18:02:13

Can you give an indication of what it was before and what it is right now on master?

4
borkdude18:02:13

the startup time

jcburley18:02:06

Oh, on my machines it varies, from 2.3x or so up to 7x to do -e (first [1 2 3]).

jcburley18:02:21

32ms on my Ryzen 3 down to 5ms. All approximations, of course.

borkdude18:02:32

impressive!

jcburley18:02:00

Still about 2.5x slower than an echo-like program written in Go, and I'm not sure how much of that is reducible in Joker vs libs vs Go itself...but thanks, yes, I'm happy with the results!

borkdude18:02:45

when this lands in a release, I'll try it out on my own machine and reflect this in a slide deck where I compare various Clojure scripting options

borkdude18:02:02

right now I've taken timings from a blog post by Stuart Sierra

jcburley18:02:20

Last I checked it was slower than bb on my Macbook, but not by as much as 2x. Haven't tried out babashka on my Ryzen 3. On my email server, an Ubuntu VM, even my wife could see the difference in startup time!

borkdude18:02:12

if you have a macOS binary around with these speedups, I'd be happy to give it a spin

jcburley18:02:13

Not handy right now, but could get one to you in the next hour or so...but how?

borkdude18:02:36

as a zip file here on Slack maybe?

borkdude18:02:59

I build binaries on CircleCI where I can get download links to the artifacts. I also sometimes just pass those links to people

jcburley18:02:21

Gmail has strict attachment size limits. Will try here.

jcburley18:02:20

Or, hey, could just post to my own web server and send you a link! 😀

jcburley18:02:33

Gimme an hour or less.

borkdude18:02:06

no hurry, the presentation is next week

jcburley19:02:46

Ooh, what presentation??

borkdude19:02:45

about babashka

jcburley19:02:12

Cool! Please share a link or something here. Is it a public event or similar?

borkdude19:02:45

it is the ClojureD conference

jcburley19:02:00

Here are some executables: https://burleyarch.com/joker/exe/ I’ll try to keep these up to date with master. Windows executables probably won’t land until Sunday evening.

borkdude19:02:52

it seems that the fast one is 5 MB bigger, is that because of more pre-compiled code?

jcburley19:02:48

Yes, that’s a 25-33% increase in size or so, pretty consistent across my 3 architectures IIRC.

borkdude19:02:58

===> multitime results
1: ./joker.fast -e "(+ 1 2 3)"
            Mean        Std.Dev.    Min         Median      Max
real        0.017       0.002       0.015       0.016       0.039

borkdude19:02:04

that's about what I get with bb too

borkdude19:02:21

for reference, I ran:

$ multitime -n100 -s0 ./joker.fast -e '(+ 1 2 3)'

borkdude19:02:03

so I guess I can just say: bb and joker have comparable startup time 🙂

👍 4
jcburley19:02:52

Well that’s pretty good (for Joker) I guess! But bb has all the JVM libs available to it? Is there a list of “big” stuff missing in bb versus Joker, that we can compare to a similar list for Joker?

jcburley19:02:56

At quick glance that seems more comprehensive than Joker…is there anything Joker has that bb doesn’t offer?

jcburley19:02:19

Also I would appreciate your input on this thread, starting with my comment the other night:

borkdude19:02:05

I think joker has a few things that bb has not, like a built-in http client, xml and yaml, which are on the list for bb.

borkdude19:02:24

we do have support for a user-land clj-http library

jcburley19:02:25

tl;dr: We can improve joker, we can improve bb, etc., but what is the 30K-foot overview of what the most promising combination of implementation language likely to be?

jcburley19:02:39

Does bb do multithreading better than Joker?

borkdude19:02:57

bb offers real threads like you're used to from clojure, it has pmap, future, agents, promises, etc

jcburley19:02:52

For classic scripting, that’s probably fine; but there are various use cases I could imagine for Joker, even as a scripting language, that bb would be better suited for simply because it has the real Clojure-style MT.

borkdude19:02:57

I think my life of adding stuff from the clojure eco-system inside bb is much easier: if I'm lucky, I can just stick a function inside a map and I'm done

jcburley19:02:39

That’s great! Offhand, do you have a sense of how GraalVM is doing in terms of stability, performance improvements, roadmap going forward, etc.? I was following the mailing list for it for awhile until early last year when things got real busy.

borkdude19:02:38

GraalVM has LTS releases and non-LTS releases, so in that regard they try to offer more stability I guess

borkdude19:02:03

they also have a Slack where you can ask questions to users and developers. they are quite responsive to issues on Github as well

jcburley19:02:57

As you’ll see from that discussion in the Issue on main programs in Joker, I’m very interested in whether Go, or GraalVM, or Rust, or whatever, would be enough-better, in terms of improvement over where we are right now, to justify a) continued work on Joker (in particular I have my gostd branch to auto-support the Go std libraries plus arbitrary 3p ones, but much more work is needed there), b) rewriting/rehosting Joker to another language, c) leaving Joker as a quick linter and moving to something like bb, or d) some other possibility I haven’t considered.

borkdude19:02:04

babashka is basically split up into two libraries: sci (the interpreter) and edamame (the code parser). other users can make something like bb but geared towards more specialized goals, like generating static websites. one such tool is called bootleg: https://github.com/retrogradeorbit/bootleg, it has hiccup, templating, etc.

jcburley19:02:22

(Right now, even without MT, Joker plus gostd would meet most of my needs beautifully.)

borkdude19:02:52

I didn't quite understand the thread about -main. I do support -main functions in bb just like the clojure cli does

borkdude19:02:39

bb also understands classpaths as produced by clojure so you can use some normal JVM libs without any change: https://github.com/weavejester/medley is an example of this

👍 4
jcburley19:02:40

I’ll make some time to look over the bb ecosystem soon then, as it came to my notice many months ago (probably due to one of your tweets) and I’m intrigued by the idea of a “more-Clojure-like-Clojure” that starts up very quickly and doesn’t have GB of VM overhead like Clojure does for even a simple service.

jcburley19:02:49

That sounds great!

borkdude19:02:46

joker is an enormous inspiration for both clj-kondo and babashka so I do follow what happens in this ecosystem too. I hope we can benefit from each others work.

🙂 4
jcburley19:02:59

If Joker had gostd (i.e. ready access to the Go std libs), drop-in core libraries (which I think we can support without too much work), and maybe “true” MT, would you see it as being worth using versus where you see bb headed?

borkdude19:02:55

I'm not familiar with the Go ecosystem at all. I see it as a plus that I get to write Clojure while writing bb and also while using it 🙂

👍 4
jcburley19:02:51

Oh, just saw your response about -main — I pointed to the part in that thread where I respond to @roman.bataev’s query about a potential Joker compiler and explain why I’m not sure that’s worth doing, even though I think about it a lot and would surely enjoy working on one. Could be in that “we can do it, but should we?” category.

borkdude19:02:58

the sci interpreter (which bb is based on) also runs on JS btw, there are already some projects leveraging this for e.g. Slack bots in a Google cloud function

👍 4
borkdude19:02:28

ah, about compiling: I think it's nice to not have to compile, for scripting, like you do with bash. Just text, no steps in between

👍 4
jcburley19:02:37

It’s funny, now that I’ve worked on Joker pretty much exclusively (during my free time) the past 18 months or so, my Clojure knowledge has probably declined somewhat since I’m not actually doing that much programming in it. I.e. most of my Clojure programming is Joker code in support of building and testing Joker, not real-world stuff. Haven’t reviewed my Clojure flashcards in over a year….

borkdude19:02:19

there is a place for everything I guess. CLJS does compilation to JavaScript which then can be quite fast on Node regarding startup, but the step in between is what it makes less attractive for scripts imo

👍 4
jcburley19:02:07

Yeah, I think @roman.bataev is most interested in behind-the-scenes, versus “overt”, compiling. I.e. not compiling a script or lib/namespace down to an artifact, but JIT’ing interpreted code in the background. Whereas I’d like to have the static-compile-to-an-artifact feature more, I think!

borkdude19:02:34

for the latter I think GraalVM is very well suited

jcburley19:02:57

Oh, interesting! Somehow I thought GraalVM was primarily a JIT.

borkdude19:02:42

GraalVM comprises several parts: Truffle/Polyglot and native-image. I've only used native-image. What it does: translate .class files to native binary. Once it's compiled there is no JIT anymore.

👍 4
jcburley19:02:04

Really gotta climb up the stack, now that the fast-init stuff is done (and after I get my taxes done 😉 ), and learn more about all these technologies at an architectural level soon.

borkdude19:02:25

if you have any questions, feel free to reach out. congrats on the startup!

👍 4
jcburley19:02:50

Thanks, will do! And have a great presentation next week!

borkdude19:02:59

there's also a #graalvm channel and some docs on graalvm here: https://github.com/lread/clj-graal-docs

👍 4
borkdude20:02:03

thanks!

👍 4
borkdude15:02:08

I'm discussing something regarding reader conditionals in the #clojure channel which may also be of interest to joker. Notably, this just got added to https://clojure.org/reference/reader#_reader_conditionals: > Implementors of non-official Clojure platforms should use a qualified keyword for their platform feature.

👍 12
Candid05:02:53

Thanks for the heads up! I can change :joker to something like :joker/clj, but I don't think there is much practical value in doing so. I mean, how many Clojure platforms (with reasonable adoption to even care about things like this) are we going to have to worry about conflicts? Did you end up changing :bb to :bb/clj?

Candid05:02:23

and :joker/clj is as much susceptible to conflicts as :joker is. Someone could as easily create a new platform with feature :joker/clj. The same goes for :bb/clj vs :bb.

Candid05:02:47

So I am leaning towards just leaving it as is.

jcburley06:02:53

IMO the advice to use a qualified keyword was less than compelling, given that it came with no guidance as to what the qualifier should be, nor any mention of grandfathering in existing unofficial implementations such as Joker and babashka. So, FWIW, I too support leaving things as-is.

jcburley06:02:53

That being said, I think it’d be unwise to name any platform feature with the cl prefix until/unless it becomes “official”. E.g. while :joker is reasonable to “stake out” for a program named “joker”, and even bb for “babashka”, I think the documentation could — instead of asking for some kind of qualified keyword — simply “reserve” all :cl* (as in a wildcard, i.e. :cl as a prefix) to official implementations.

borkdude07:02:05

I didn't change it as I also didn't want to break existing scripts out there.

borkdude07:02:44

Thanks for you feedback!

borkdude13:02:28

We can probably take this to the International Unofficial Clojure Implementation Summit, together with Clojerl 😉

🙂 4
borkdude15:02:30

To be honest, I have no clear idea on what to name the feature. See #clojure .

👍 4
jcburley18:02:31

This was merged by @roman.bataev last night (thanks!!). Please try it out, by pulling down the latest master, building, and testing it. (It’s working great for me so far on my email server, where it’s noticeably faster starting up.)

jcburley18:02:18

I should mention (since I got bit by this myself just now, during a demo) that, to install the fast-startup version of Joker, after running run.sh or go generate ./..., please do:

$ go install -tags fast_init
Or just copy joker.fast to wherever you want it, naming/linking it "`joker`".

jcburley18:02:00

Still about 2.5x slower than an echo-like program written in Go, and I'm not sure how much of that is reducible in Joker vs libs vs Go itself...but thanks, yes, I'm happy with the results!

jcburley18:02:20

Last I checked it was slower than bb on my Macbook, but not by as much as 2x. Haven't tried out babashka on my Ryzen 3. On my email server, an Ubuntu VM, even my wife could see the difference in startup time!