Clojurians Log v2

pp builds on pr-str it seems, so no way that will be faster

if you optimize for performance/serialization, maybe choose transit or nippy?

I could, but I reeeeeally want the data at rest to be EDN....

well I guess I could at least check what the difference is

Maybe you can write to a tempfile with a writer instead of to a big string that is sent to the pod?

this isn't just about bb I think @jeroenvandijk

pr is quite inefficient - for each nested object, str is called. And that constructs a new string builder every time. If you really need the intricacies of printing with its :type support and all the multimethods, there's unlikely to be a way around it short from copying all that functionality and overriding str usage with something that reuses the same buffer. If you need a small subset of pr functionality, and a known set of types, rolling out your own EDN serializer should be quite easy.

Huh I'm a bit surprised at these numbers, but nippy freeze is actually comparable to pr-str? and thaw is slower than fast-edn. It does make a smaller output though. But on the whole it doesn't seem like it'd be faster.

Writer comparison on the same value:
  pr-str-meta          243.1 ms
  fipp (with meta)    8922.7 ms
  pp (no meta)        2284.6 ms
  nippy/freeze         155.8 ms
  (nippy bytes:      6.1 MB)
  nippy/thaw           176.2 ms

@p-himik thanks for the tip, maybe I'll look at doing that

or perhaps we could change that StringBuilder problem in Clojure? @alexyakushev may have some thoughts on it

i ran into this problem when writing a pretty printer for the Fluent file format last year. i ended up writing my own wrapper around StringWriter/.write and then manually calling (str ...) myself and writing each character by hand.

went from ~6ms to write a 2k file to 500us, with a 1-to-1 java version running in 250us.

print -> binding -> apply pr -> pr-on -> print-method -> multimethod dispatch -> Writer/.write

that's a lot of steps if you already have a string

When in doubt, profile. Or I can do it for you if you post a sample EDN file to roundtrip.

But from the sound of it, 150ms for 20MB edn doesn't sound abhorrently bad, tbh. You may expect 2x improvement from here but I doubt more is possible.

well I wouldn't say no to expert advice 😄 the same edn I'm using is this, I think I got it from a roam book club

what really made me go "huh" was how fast-edn made the edn parsing faster than the writing, which made me wonder if I was missing something obvious

in my head going from a 20mb str to in-memory data structures should be slower than the reverse

Context is that I'm trying to optimize https://github.com/filipesilva/sqlatom

you could maybe try https://github.com/bsless/prrr

Binary storage like nippy or deed or the Java freeze library would be better, I guess

> Just an idea: why not storing data not in EDN format at all? I understand this is an option and it's just one I'm not very interested in right now. I care a lot about being able to read the serialized data with low ceremony. I guess I'm coming from the markdown camp here, I like the at-rest data format to be acessible. It also somewhat lessens the incentive that on that particular 20mb case, it actually looks like the nippy round trip should be slower than pr-str+fast-edn/read-string, at least according to the naive attempt I made in https://clojurians.slack.com/archives/C03S1KBA2/p1777376552815469?thread_ts=1777375536.274579&cid=C03S1KBA2

let me double check the nippy case using criterium so it's a bit more reliable...

Could you please also take deed into account? When measuring it agains nippy, this is what I get:

(quick-bench
      (do (deed/encode-to-bytes STRESS-DATA) nil))
Evaluation count : 4152 in 6 samples of 692 calls.
             Execution time mean : 149.130920 µs

(quick-bench
      (nippy/freeze STRESS-DATA {:compressor nil
                                 :encryptor nil}))
Evaluation count : 1980 in 6 samples of 330 calls.
             Execution time mean : 313.618046 µs

Apologies if this is a dumb question, but since you made deed you probably have a good answer... how does deed handle reader information? I think I don't understand this well myself tbh. I know inst and uuid are built in. But I think that if I add some custom ones (e.g. datascript/DB) onto my process, I should be able to use pr-str to print edn that says "this should be read back with these readers". Does deed "just" handle that too?

another benchmark: I prepared a map and dumped it into two forms: deed bytes and pr-str string. Then:

(quick-bench (deed/decode-from deed-bytes))
Execution time mean : 22.085057 µs

(quick-bench (edn/read-string -s)) ;; fast-edn, not default
Execution time mean : 45.662700 µs

> I know inst and uuid are built in. Just in case, careful with inst - multiple different things can be printed as #inst "...". Meaning, it's not round-trippable in general.

> how does deed handle reader information? Deed uses binary format, it has nothing in common with readers. The UUID type is stored like (short 0x0042)(long high_bits)(long low_bits) and gets read back using that logic

deed is like nippy but with some improvements

i was surprised to see fipp perform so slowly compared to the others

@dpsutton you shouldn't dismiss the possibility that my benchmark is bad 😄

@igrishaev so what would happen with deed and the custom datascript/DB reader? would it round trip correctly?

well, I don't know much about datascript but I think yes it will be alright. Deed is bidirectional. Everything you wrote gets read back with no issues

also, if you're going to store a datascript db in sqlite, why not take the standard datomic? It has a nice local file storage out from the box

this particular lib is meant to work cross process (jvm/bb) without a coordinator like a transactor, but fwiw I also played around to making datomic work kinda like a lib using sqlite lol https://github.com/filipesilva/datomic-pro-manager#library-usage

Ok I have a benchmark here that should be easy to interpret... it compares deed, with nippy, with fast-edn + pr-str , with fast-edn + fast-pr , where fast-pr is a`pr-str` that claude made from some of the comments on this thread. The large edn and fast-pr code attached to the message. Results below the code.

(ns roundtrip
  (:require [criterium.core :as cc]
            [fast-edn.core :as edn]
            [taoensso.nippy :as nippy]
            [deed.core :as deed]
            [filipesilva.sqlatom.fast-pr :as fast-pr]))

(def buf (* 64 1024))

(defn -main [& _]
  (let [data (edn/read-string
              ;; Parse edn, stripping unknown tags (e.g. <#C07V8N22C|datascript>/DB) to plain values
              {:default (fn [_ v] v)}
              (slurp "bench/roam-book-club-2026-02-18-11-31-58.edn"))]
    (println "\n=== nippy/thaw ∘ nippy/freeze ===")
    (cc/bench (nippy/thaw (nippy/freeze data)))

    (println "\n=== deed/decode-from ∘ deed/encode-to-bytes ===")
    (cc/bench (deed/decode-from (deed/encode-to-bytes data)))

    (println "\n=== fast-edn/read-string ∘ pr-str ===")
    (cc/bench (edn/read-string (pr-str data)))

    (println "\n=== fast-edn/read-string ∘ fast-pr/pr-str-fast (64KB buf) ===")
    (cc/bench (edn/read-string (fast-pr/pr-str-fast data buf))))
  (shutdown-agents))

filipesilva@m4 ~/r/p/sqlatom (master)> bb roundtrip

=== nippy/thaw ∘ nippy/freeze ===
Evaluation count : 360 in 60 samples of 6 calls.
             Execution time mean : 174.823633 ms
    Execution time std-deviation : 3.533381 ms
   Execution time lower quantile : 170.357499 ms ( 2.5%)
   Execution time upper quantile : 181.138916 ms (97.5%)
                   Overhead used : 1.090493 ns

Found 1 outliers in 60 samples (1.6667 %)
	low-severe	 1 (1.6667 %)
 Variance from outliers : 9.3625 % Variance is slightly inflated by outliers

=== deed/decode-from ∘ deed/encode-to-bytes ===
Evaluation count : 480 in 60 samples of 8 calls.
             Execution time mean : 148.642656 ms
    Execution time std-deviation : 10.025610 ms
   Execution time lower quantile : 136.283187 ms ( 2.5%)
   Execution time upper quantile : 168.793429 ms (97.5%)
                   Overhead used : 1.090493 ns

=== fast-edn/read-string ∘ pr-str ===
Evaluation count : 180 in 60 samples of 3 calls.
             Execution time mean : 340.587618 ms
    Execution time std-deviation : 14.364932 ms
   Execution time lower quantile : 316.346082 ms ( 2.5%)
   Execution time upper quantile : 373.146826 ms (97.5%)
                   Overhead used : 1.090493 ns

Found 2 outliers in 60 samples (3.3333 %)
	low-severe	 2 (3.3333 %)
 Variance from outliers : 28.6825 % Variance is moderately inflated by outliers

=== fast-edn/read-string ∘ fast-pr/pr-str-fast (64KB buf) ===
Evaluation count : 600 in 60 samples of 10 calls.
             Execution time mean : 103.030394 ms
    Execution time std-deviation : 3.839327 ms
   Execution time lower quantile : 96.120749 ms ( 2.5%)
   Execution time upper quantile : 110.588021 ms (97.5%)
                   Overhead used : 1.090493 ns

Found 1 outliers in 60 samples (1.6667 %)
	low-severe	 1 (1.6667 %)
 Variance from outliers : 23.8279 % Variance is moderately inflated by outliers

I know that datascript stores a sorted set of datums, where each datum is a vector like [e a v t op] . I think you can have a table in a SQLite DB likes this:

create table datums (
  e integer,
  a text,
  v text,
  t integer,
  op boolean
);

This ignores how big the serialized data structure is btw, deed and nippy should be making smaller stuff than edn. But the baseline of fast-edn+pr-str isn't as slow as I'd expect for "just" edn. I don't feel very confident in fast-pr, that's just something thrown together by the llm and probably falls flat on lots of corner cases, but is interesting in a "what if pr-str was faster" kind of way.

you only need to coerce v to the right Clojure type. each V might be an edn string

I could, but I think I'd be happy enough just serializing edn and not trying to do anything specific to datascript... that's kind of what I was trying to keep as the "contract" in sqlatom, that stuff was just edn, and you could put in it anything you were putting in a normal atom in your program.

https://flamebin.dev/bipRRy I take my words back, this is quite terrible allocation-wise. Expensive iteration, dynvar usage, creating maps for some reason. There's plenty of fat to trim.

@rolthiolliere I took a look at https://github.com/bsless/prrr/blob/master/src/bsless/prrr.clj and it broadly seems to be doing what Noah said he did, so I tried that fast-pr impl instead

another bench with that huge file:

(quick-bench (with-open [r (-> "roam-book-club-2026-02-18-11-31-58.edn"
                  
                  
  (fast-edn/read-once {:readers {'datascript/DB identity}} r)))
Execution time mean : 220.336296 ms

(quick-bench (deed/decode-from ( "ds-data.deed")))
Execution time mean : 197.419803 ms

whoa that's surprising

so reading edn is just... pretty fast

yeah the difference is not so hi

thanks for engaging with me on the "what if edn" side of things, I know it must be a bit frustrating because you obviously are very familiar with more advanced serialization options

well, I just wanted to note that storing data in EDN is a bit painful as it's not bi-directorinal. Say, one day somebody writes a regex or java.time.Instant and it's over: you cannot read it back. I even had to implement this weird thing: https://github.com/igrishaev/taggie

yeah I think Nikita also tackled that in https://github.com/tonsky/clojure-plus#clojureprint

yes but with some limitations. As far as I know, he doesn't extend clojure.pprint multimethods. So if you pprint data with clojure-plus, some items won't have tags.

The problem is, clojure.pprint defines its own multi-methods for arrays and other types, which taggie extends

Thanks for all the help everyone! Ended up doing a custom pr-str that mostly does what Noah said, together with fast-edn/read-edn for https://github.com/filipesilva/sqlatom. For that particular 20mb edn it resulted in a bench'ed 76ms reset! (down from 314ms), and 140ms swap! (down from 649ms). No changes for bb, the custom pr-str is wayyyy slower for bb than pr-str, I guess on account of being interpreted. Left all the benchmarking stuff in the repo. Final edn roundtrip numbers (elided for brevity):

filipesilva@m4 ~/r/p/sqlatom (master)> bb roundtrip

=== nippy/thaw ∘ nippy/freeze ===
             Execution time mean : 185.827417 ms

=== deed/decode-from ∘ deed/encode-to-bytes ===
             Execution time mean : 158.242840 ms

=== fast-edn/read-string ∘ pr-str ===
             Execution time mean : 359.510660 ms

=== fast-edn/read-string ∘ fast-pr-str/pr-str ===
             Execution time mean : 123.431958 ms

=== fast-edn/read-string ∘ fipp/pprint ===
             Execution time mean : 9.425517 sec

=== fast-edn/read-string ∘ pp/pprint ===
             Execution time mean : 1.882720 sec

I don't know if it's prettier, but you could try (str/replace s (re-pattern (str "^\\Q" prefix "\\E")) "")

(cond-> s (starts-with? s prefix) (subs (count prefix)))

@rolthiolliere I like your suggestion better 🙂

subs , that’s the function I was missing. Thanks.

@rolt Why cond-> and not a normal if?

@hbrng.computer the "meat" of my response was the use of subs. I also proposed cond-> because it could make for clearer code in some contexts but if it's a simple function body I'd definitely use if here.

I’ll attach a sample of the json file, i’ve marked the object I want to find with the comment

;; maybe find THIS

JSON here is really orthogonal. As soon as you parse it, you got regular Clojure data on your hands. So, judging by the code, what you need to to find the very first item that satisfies a pred in a collection of items that belong to another very first item that satisfies a pred. This begs two changes: • Extract "find first item that satisfies a pred" into its own function • Label things instead of deeply nesting forms Also, your member function, assuming you only use it on data that comes from JSON, is way too detailed. No need for empty? - it's done automatically by the combination of some and boolean wrapped around the whole body. I probably wouldn't even write that function and instead rely on (some #(= % target) items).

Just in case - note that your function will not find all of the items. And sometimes will fail to find the first nested match if it's nested under the second top-level match. Whether or not it's desired behavior, I don't know. But given how you left multiple markers in the JSON data, perhaps your impl is wrong.

I tend do do a lot of “find first thing that matches a predicate” in my programming. json or otherwise. i’m surprised there isn’t a build-in function for that.

I don’t understand your warning that my code might be wrong. please explain. if there’s a bug i’d like to understand it better.

> i’m surprised there isn’t a build-in function for that. As you can imagine, the core team is well aware and IIRC the sole reason for not including such a function is unwillingness to make it a common pattern. Since it's better to rely on proper data structures that support fast lookup rather than always relying on O(N), which is much more common in e.g. the JavaScript world where its arrays offer indexOf. > I don’t understand your warning that my code might be wrong. Suppose you have a nested structure like

[{:matches? true
  :items [{:matches? true}]}]

[{:matches? true
  :items []}
 {:matches? true
  :items [{:matches? true}]}]

I think I see what you mean. In the parsing I believe my filters only result in one match. I believe that because I understand the shape of the json. However, it would be nice to assert that in my code, at least to make that apparent to the person reading the code.

How’s this? ---not yet tested---

(first
     (for [as (:assignments (get-json-data json-file-name))
           :when (= assignment (:slug as))
           :when (member login (:logins (:subscriptions as)))
           :let [submission (:submissions as)]
           :when (= (str assignment "-")
                    (:tagPrefix submission))]
       (:closingAt (:dates submission))))

perhaps I need a function (in my local library) called first-and-only which asserts there is not another one I’m missing?

The :let [...] should be just .... But yeah, that's probably how I'd write it. first-and-only also makes sense, although I'd probably name it the-only! and (throw ...) instead of (assert ...).

doesn’t the ! normally specify that something is modified?

I think you’re right about the :let […] … good eye

isn’t there some other idiom for (:key1 (:key2 (:key3 … hash)))

ahhh get-in

! generally means "expect side effects". Not necessarily modifications. Check out e.g. clojure.core/io! or clojure.core/run! or clojure.core/volatile!.

When keys are keywords, I usually prefer (-> m :key3 :key2 :key1).

aside: member could be (contains? (set items) target)

Quite a lot of extra work though. :) And not shorter than (some ...).

actually part of the motivation (long ago) of writing this member function was to avoid evaluating even one-too-many predicates. At the time I had predicates which were very expensive to compute. each successive evaluation was double the compute time as the previous. That’s way member carefully calls reduce rather than filter and friends.

thereafter, i’m just copied the function into all successive programs where i needed the function.

some also short-circuits and doesn't evaluate the predicate more times than necessary. No need to copy it around if you never use it - if you rely on contains? for sets and some for every other collection. :)

> my clojure code often seems to be written in the reverse order and other programmers I tend to prefer using ->> or -> (or even as->, when mixing positions) here:

(->> data
     :assignments
     (first-and-only! match-fn)
     :submissions
     etc...)