Clojurians Log v2

Network time might dominate, but it doesn't result in a perceptive performance degradation. Even if there's a progress spinner, it doesn't prevent a user from otherwise interacting with an app. But de/serialization actively prevents it - if it's longer than 16 ms it's a skipped frame at best and a prolonged lack of interactivity at worst. That's not a hypothetical, that's what I already have to deal with.

Let’s bring the discussion back to numbers- - Using Node for Cljs - With key deduplication - And about 106kb of random edn data https://github.com/taoensso/sente/blob/0431ca86dbe43386a0e5d10416804891fec580a1/test/taoensso/sente_tests.cljc#L38 on your example - Running 100 laps I’m seeing the following on my system:

Clj:  {:edn {:time 720,  :size 110030}, :transit {:time 132, :size 81782}, :msgpack {:time 107, :size 44676}}
Cljs: {:edn {:time 2574, :size 114652}, :transit {:time 346, :size 86404}, :msgpack {:time 520, :size 49523}}

Clj  MessagePack vs Transit: MP uses 0.55x the size and 0.8x the time (~1.3ms per lap)
Cljs MessagePack vs Transit: MP uses 0.58x the size and 1.5x the time (~5.2ms per lap)

Clj  MessagePack vs Edn: MP uses 0.40x the size and 0.15x the time (~1.3ms per lap)
Cljs MessagePack vs Edn: MP uses 0.43x the size and 0.2x  the time (~5.2ms per lap)

I should add- I’ve not worked much with Web Workers, but if you’re dealing with large payloads and are very sensitive to processing time, that might be another option? Sente v1.21's packers are now async, so I suspect wrapping any packer (edn, Transit, or MessagePack) to process on a worker shouldn’t be too difficult.

Awesome, thanks. Yeah, at that point I wasn't arguing against using msgpack with Sente by any means, just describing where my initial concerns came from. 1.5x actually sounds significantly better than what I remember. Web workers could be interesting, but IIRC people have experimented before and they were a net negative since CLJS data structures aren't transferrable.

> 1.5x actually sounds significantly better than what I remember. Just to clarify: MessagePack is just a format. Performance depends very much on the implementation and this is a new implementation so prior benchmarks won’t really be relevant.

Indeed, and it makes me hopeful.

Interesting. Does msgpack really beat the performance of transit+json? I remember doing some experiments years ago and the latter was winning by a landslide, probably due to the JSON part being already heavily optimized. But it was a long time ago and there's always a chance I did something wrong or simply misremember things. Or is it not the performance you're concerned about when choosing what the default will be?

I had the same question. I ran benchmarks a year or two ago and IIRC, there was no scenario where MsgPack beat JSON (using the Java MsgPack implementation). Also, it seems to me that taoensso.msgpack.impl reflects, no? Perhaps there's some low-hanging fruit there to pick?

So here’s a quick/rough comparison on the Clojure side:

(comment
  (require '[taoensso.encore :as enc])
  (require '[taoensso.sente.packers.transit :as transit])
  (require '[taoensso.sente.packers.msgpack :as msgpack])

  (let [ep          edn-packer
        tp (transit/get-packer)
        mp (msgpack/get-packer)
        data {:a :A :b :B :c "foo", :v (vec (range 128)), :s (set (range 128))}]

    {:speed
     (enc/qb 1e4 ; [1040.5 262.6 151.74]
       (i/pack ep nil data (fn [x] (i/unpack ep nil (get x :value) (fn [y] (get y :value)))))
       (i/pack tp nil data (fn [x] (i/unpack tp nil (get x :value) (fn [y] (get y :value)))))
       (i/pack mp nil data (fn [x] (i/unpack mp nil (get x :value) (fn [y] (get y :value))))))

     :size ; [841 872 298]
     (mapv count
       [(i/pack ep nil data (fn [x] (get x :value)))
        (i/pack tp nil data (fn [x] (get x :value)))
        (i/pack mp nil data (fn [x] (get x :value)))])})

  {:speed [1040.5 262.6 151.74], :size [841 872 298]})

on the Clojure sideAh, I was comparing in CLJS, across browsers. I can always beef up my server, I can't do the same for my users. :) > • MessagePack beats Transit size by about 2.9x and edn by about 2.8x Have you tried with a more representative sample, with lots of relatively long but repeated keywords?

I can always beef up my server, I can’t do the same for my users.Not sure how much client-side performance matters in practice TBH, aside from data size. That’ll rarely be a bottleneck in practice. Server-side efficiency is more interesting IMO. You can of course always throw more money/servers at the problem, but there’s also an incentive to not do that if it can be avoided. > Have you tried with a more representative sample, with lots of relatively long but repeated keywords? Problem is that different applications will have different representative samples. Would recommend testing with payloads that reflect your particular application. Also a lot of applications/folks won’t care about the performance or size delta one way or the other. If your db takes a second to reply to a request and you’re sending tiny payloads, shaving 60 bytes off a network request probably won’t be meaningful.

> Not sure how much client-side performance matters in practice TBH, aside from data size. That’ll rarely be a bottleneck in practice. We definitely have different practice. I habitually transfer MBs of data around via Sente connections. Also, battery life. > Problem is that different applications will have different representative samples. Right. But {:a :A ...} is unlikely to be representative anywhere... Yet another factor is observability. With transit, I can view the data via the Network tab even in production, even when it's not verbose. Using msgpack by default would demand extra configuration for something that now just works.

On a random sample of real data:

(let [mp (msgpack/get-packer)
      tp (transit/get-packer)
      data (read-string (slurp "/home/p-himik/tmp/data.edn"))]
  (mapv count
        [(i/pack mp nil data :value)
         (i/pack tp nil data :value)]))
=> [177095 133222]

Typical users switching from edn should see an improvement. Whether that improvement would matter depends on the application + details. Typical users switching from Transit may also see an improvement, with the same proviso. > We definitely have different practice. My point is that a 20% serialization efficiency improvement on the client is unlikely to make a difference that is perceptible to most users in typical applications. I’d expect battery life to be impacted more by a reduction in data size, but again YMMV. Details will determine a lot here. > Right. But {:a :A ...} is unlikely to be representative anywhere... This is a synthetic example sketched out quickly since you asked for some numbers. You’re welcome to tweak with data that you find reasonable for your use case. > Yet another factor is observability. With transit, I can view the data via the Network tab even in production, even when it’s not verbose. Using msgpack by default would demand extra configuration for something that now just works. I get the impression you somehow think I’m advocating for you changing your setup? You’re of course welcome to stick with Transit if you’re happy with Transit. Transit support isn’t going anywhere. Recent changes have increased support for a wider range of packer types (incl. async). This is an additional option for folks that might find it beneficial.

> On a random sample of real data Could you share an example of what some of your data looks like?

> Typical users switching from edn should see an improvement. Ah, wait, the EDN was the default? Oh, sorry for the noise then, for some reason I thought it was already transit+json. > This is a synthetic example sketched out quickly since you asked for some numbers. Alright, cool. I thought it was copy-pasted from an actual benchmark that affected the decision. > Could you share an example of what some of your data looks like? Most of the payload is a somewhat nested structure of maps with some leaves being large vectors of uniform maps. Repeated keywords get nicely compressed by transit. I'll try creating a representative but an anonymized sample.

Performance on my data:

(let [mp (msgpack/get-packer)
      tp (transit/get-packer)
      data (read-string (slurp "/home/p-himik/tmp/data.edn"))]
  (enc/qb 1e2
          (i/pack mp nil data #(i/unpack mp nil (:value %) :value))
          (i/pack tp nil data #(i/unpack tp nil (:value %) :value))))
=> [489.53 350.04]

> Ah, wait, the EDN was the default? Yes, edn is currently the default 👍 > Repeated keywords get nicely compressed by transit. That’d explain the difference in your result. The current MessagePack implementation doesn’t do any kind of optimization for this. Adding something like that would be pretty easy if there’s interest.

Oh, here's a vector for 1e4 that I gave up upon: [47541.77 34988.93].

Here's anonymized but still representative data. Keywords that are the same across the data have stayed the same and mostly retained their length, same for strings.

De/serialization performance and size of highly repetitive data like this is going to be dominated by whether you have a key(word) deduplication mechanism or not. Neither edn nor Sente’s MessagePack currently has a dedupe mechanism, though adding one should be pretty easy.

Which would also affect de/serialization performance. But maybe negligibly so, I can't tell. FWIW I myself would probably not touch EDN at all. I don't know of any redeeming qualities as compared to the other formats, except for copy-paste'ability. But that goes out the window with deduplication, so it feels that if someone deliberately chose EDN then it was exactly because it's readable and copy-paste'able, and that someone thus would never enable deduplication. BTW, just in case - deduplication is useful even when a particular message doesn't have repetitions at all. I don't remember if Sente does it (maybe it's something that happens implicitly and by default due to the way transit is used there), but it's possible to reuse the same transit keyword cache across calls to de/serialization functions. So as long as there's some repetition, the mechanism could be useful.

Which would also affect de/serialization performance. But maybe negligibly so, I can’t tell.For each duped value, you can cache the de/serialization and write a 1-2 byte placeholder instead of the actual payload. That adds up quickly to both space+time savings. > and that someone thus would never enable deduplication. To clarify: a dedupe mechanism would be relevant for the MessagePack-based serialization, not really for edn. > deduplication is useful even when a particular message doesn’t have repetitions at all. That’s a bit of a different scenario, with different tradeoffs. Caching within a particular payload is pretty easy without affecting the API. Caching across payloads means that you either need to make your serialization stateful (which is especially difficult in the client+server context) or you need to have out-of-band config. In a case like yours out-of-band would offer the best time+space, but then the API needs to support that and it stops being plug-and-play. Alternatively some sort of simple compression (e.g. gzip or Brotli) could help (and is now possible with Sente). But even with compression you’d still see a benefit from key deduplication in cases like yours. I.e. best case would likely be something like dedupe+Brotli. Short tl;dr: per-payload key dedupe would offer significant time+space savings in cases like yours, wouldn’t affect the API, and should be pretty simple to implement. Nippy already has something similar, though it’s not automatic.

So I get the following results with https://clojurians.slack.com/archives/C06MAR553/p1756817634095959?thread_ts=1756808512.361059&cid=C06MAR553 and a quick key caching implementation:

;; [edn transit msgpack], 1e2 runs
{:speed [11430.51 2293.22 2391.8], :size [204808 133275 82658]}

Oh, cool! My use-case is predominantly dictated by CLJS though, and I haven't tested it myself with that data.

Cljs size would be identical, would need to measure the time. But as mentioned - network typically takes longer than de/serialization, and uses relatively more energy (battery). I.e. payload size typically dominates the user experience, which is why even reasonably expensive compression is often a net win.

True, but the page can do something else while the data is being received over the network. The page cannot do anything while the data is being de/serialized.

The network time will be the same between a low- and a high-end device. But if de/serialization is relatively slow, the low-end device will suffer incomparably more.

I’d expect the realistic variation in data transit time to be significantly larger than the realistic variation in device performance - especially for something relatively so inexpensive. If you’re transferring so much data that de/serialization time is actually going to be perceptible to the user, then you’re transferring a lot of data - and I’d expect network time to dominate. Anyway, I’m of course happy for folks to use whatever packer strategy they prefer - and to optimise for whatever they prefer / makes sense for their use case.

Update, with a little tuning:

;; Absolute
Clj:  {:edn {:time 673.23, :size 109957}, :transit {:time 126.03, :size 81709}, :msgpack {:time 101.61, :size 44638}}
Cljs: {:edn {:time 2563,   :size 114703}, :transit {:time 346,    :size 86455}, :msgpack {:time 350,    :size 49515}}

;; Relative to edn
Clj:  {:edn {:time 1.0, :size 1.0}, :transit {:time 0.19, :size 0.74}, :msgpack {:time 0.15, :size 0.41}}
Cljs: {:edn {:time 1.0, :size 1.0}, :transit {:time 0.13, :size 0.75}, :msgpack {:time 0.14, :size 0.43}}

;; Relative to transit
Clj:  {:edn {:time 5.34, :size 1.35}, :transit {:time 1.0, :size 1.0}, :msgpack {:time 0.81, :size 0.55}}
Cljs: {:edn {:time 7.41, :size 1.33}, :transit {:time 1.0, :size 1.0}, :msgpack {:time 1.01, :size 0.57}}

Awesome!

FYI: its XDG_CONFIG_HOME https://practical.li/blog/freedesktoporg-xdg-standard-for-clojure-development-tools/#xdg-environment-variables

that's wild! very cool, thank you

that's cool thanks @alexyakushev - I'm completely happy with this convention, but an expected question that I've had when introducing global config files is that people either like to put it in ~/.config/nrepl or respect XDG_CONFIG. So just a heads up

Right now it references a custom env var NREPL_CONFIG_DIR, but I agree that XDG_CONFIG makes much more sense. I think we can change that without breaking much.

When/where does it actually read that EDN file? If I start a plain nREPL or CIDER-enhanced nREPL with 1.4.0, I get the reflection warning as expected. If I use -f rebel-readline.main/-main to start nREPL (i.e., start Rebel Readline via nREPL), I do not get the reflection warning. (but I confirmed setting NREPL_CONFIG_DIR=$HOME/.config/nrepl works to allow you to follow XDG conventions, thank you!)

@seancorfield Are you sure you don't start a plain rebel-readline? It should be rebel-readline.nrepl.main if my config is correct.

Well, my current approach does start a CIDER-enhanced nREPL server as well as the interactive REPL... so maybe the global var only affects the server in this setup? Let me look at Rebel Readline in more detail (is that ns a new addition?).

Ah, yes: Recent updates include nREPL support. For details, refer to rebel-readline-nrepl.

Er, no, that's just Rebel Readline as a client for nREPL. That's not the same thing as I'm currently doing (a single command to start both, instead of starting nREPL and then starting the Rebel client) which is why I didn't switch to that when it was added.

If you don't use the new namespace (`rebel-readline.nrepl.main`), then I don't think you are connecting to an nREPL server with Rebel. Which means you start a separate Rebel REPL which is not related to nREPL, and thus it doesn't read its config, naturally.

Same goes for any plain REPL, for example if you just start clj – no reflection warnings will be enabled there, unfortunately.

But using -f does start an nREPL server. And, yes, if I then connect a client to it (like Calva), I get reflection warnings on evals from the editor. I was hoping -f used the same context as the nREPL server -- although I already have to use -e (apply require clojure.main/repl-requires) so I guess that speaks to the difference in context. Fair enough.

Hm, I understood you wrong, but I think I got it now. You start via -m nrepl.cmdline -f rebel-readline.main/-main and the warnings aren't there? I should investigate that. Can you please paste the complete command that you use?

Admittedly, even in this setup, having reflection warnings for all editor-connected evals is useful, and it doesn't make my interactive REPL worse so it's a net win overall.

Well, the command is constructed dynamically, based on what's on the classpath. It's all in my dot-clojure repo. So if various things are on the classpath, it adds extra middleware etc.

I can reproduce this:

$ clojure -A:nrepl:rebel -m nrepl.cmdline -f rebel-readline.main/-main
user=> *warn-on-reflection*
false

I will look into how -f works, apparently it is indeed different somehow.

Yeah, that's close to what I'm doing. I have -i in there as an argument still for nrepl.cmdline but I guess it ignores that if -f is given?

@seancorfield I figured it out. Passing -f rebel-readline.main/-main doesn't magically make Rebel connect to the nREPL server. It starts its own REPL. So you get two REPLs really – the nREPL server (that you connect to with Calva or whatever else), and the Rebel REPL.

Yeah, I had just hoped that by the time Rebel created its REPL, the Clojure context would already have the reflection var set globally 🙂

Here is the thing, dynamic vars cannot really be set "globally", unless you alter-var-root them which is kinda harsh. I mean, it still can be done, but then it is probably not nREPL's job to perform such drastic measures. And even then, that would only change dynvar roots but not the individual thread bindings. All in all, dynamic variables is a quite confusing mechanism, it's not a simple switch. And it would probably be better if it was a single mutable toggle.

> unless you alter-var-root them which is kinda harsh I think for nREPL that would be fine to be honest ;)

Yeah, I get it. I just wasn't quite sure of the context for -f since I haven't dug into nREPL's source much. Maybe with Clojure 1.13 we'll be able to global set this from the command-line etc 🙂

@borkdude I've just checked with simple clj:

user=> (.bindRoot #'clojure.core/*warn-on-reflection* true)
nil
user=> *warn-on-reflection*
false

I know. It can be done with a user.clj + alter-var-root. But yeah, when evaluating in the REPL, Compiler/load already has the first binding from clojure.main probably. A solution would be to make your own clojure.main ;P

@seancorfield If you are interested, I managed to make it work with -f and Rebel. This is my deps.edn alias:

:nrebel {:extra-deps {global/user-clj {:local/root "/Users/myuser/.clojure/"}
                      com.bhauman/rebel-readline {:mvn/version "0.1.5"}
                      com.bhauman/rebel-readline-nrepl {:mvn/version "0.1.6"}}
         :main-opts ["-m" "nrepl.cmdline" "-f" "nrebel/connect-to-nrepl" "-i"]}

(ns nrebel
  (:require rebel-readline.nrepl.main))

(defn connect-to-nrepl [host port options]
  (rebel-readline.nrepl.main/start-repl {:port port}))

$ clojure -M:nrebel
nREPL server started on port 57761 on host localhost - 
[Rebel readline] Type :repl/help for online help info
user=> *warn-on-reflection*
true

Wow, okay... I'll bookmark this and have a play with my dev/repl stuff in dot-clojure later!

> Right now it references a custom env var NREPL_CONFIG_DIR, but I agree that XDG_CONFIG makes much more sense. I think we can change that without breaking much. Yeah, that’d be most a matter of tweaking:

(def config-dir
  "nREPL's configuration directory.
  By default it's ~/.nrepl, but this can be overridden
  with the NREPL_CONFIG_DIR env variable."
  (or (some-> (System/getenv "NREPL_CONFIG_DIR") io/file)
      (io/file home-dir ".nrepl")))

I think originally I didn’t consider a global config, so that’s probably why I didn’t add a check for XDG_HOME or however that env var was named.

@alexyakushev Thank you for the connect-to-nrepl trick you posted -- I integrated it into my :dev/repl setup (my dot-clojure repo) and it works brilliantly!

I have not.

Pinging @jasonbell here since the re-release of this lib sprung forth from this conversation: https://clojurians.slack.com/archives/CBJ5CGE0G/p1756674115749719

I also recommended ChronicleQueue as a shot in the dark in that thread ;)

Any extra background as to why that project is earning multiple “in the dark” shots? 🙂

On paper it sounds good but have no experience with it

Presumably, it’s good; but ever since I started with Clojure my default is “not everything is awesome” 😅

Which can sometimes overcorrect but more often than not has been a good default, lol

The folks from that team seem to know what they're doing. Big "If I swizzle this pointer wrong my employer, a High-Frequency Trading firm, is going to crash and I'll be banished from the industry" vibes. Again, have not used it in anger, just read a lot about it and other works from that group.

Asking, because I did use durable-queue a long way back for a small-ish use cases, and it did what it promised at the time; again that was a long way back

> The folks from that team seem to know what they’re doing. Ah, nice. That’s good context.

I'm sure durable-queue is going to be an excellent choice for anyone interested in "durable (but-on-a-single-machine) queues" reading about it in Clojurians.

> If I swizzle this pointer wrong The Swizzling Dark Arts 😅

The JVM does not have true swizzling, does it?

Sure it doesn't 😉

I recall doing this in Objective-C, or more accurately, my teammate doing it. Worked like… magic! 👻

> Sure it doesn’t Are you implying it does, via some super weird trick? 😄

Like… idk… forking the JVM?

Ofc, if we’re going to call to C/C++, anything is possible, for better or worse.

There are indeed many super-awesome-sounding repos in here https://github.com/OpenHFT

Nice, I somehow missed this library when I needed something similar recently. Do you happen to know offhand if if it's meant to be safe to use across multiple JVMs on the same machine? (my guess would be probably not, but I'm not sure)

No, it is not designed for coordination between multiple processes.

Shot in the dark, you might be looking for something like https://github.com/OpenHFT/Chronicle-Queue @jjttjj

ChronicleQueue looks interesting! @joe.lane have you used it?