Clojurians Log v2

+1 transit. It is edn for the wire.

I've started with EDN, in the past, then tried transit when EDN became a performance issue. Transit was not compelling, perf-wise, compared to just JSON. I ended up expanding on the excellent jsonista library to add support for "tagged" JSON, which supports keywords, sets, and so on. This is significantly faster than both EDN and Transit.

The docs are here: https://github.com/metosin/jsonista?tab=readme-ov-file#tagged-json

Just dug up the PR. Looks like the perf wins over Transit were 40-90%: https://github.com/metosin/jsonista/pull/36 Maybe interesting for others currently using Transit. 🙂

It's pretty common to do this. If performance is an issue transit may also be worth trying

We did this at my last job but I didn't set it up so it mostly "just worked"

Well to be more specific we used transit

When Transit came out, one of the motivations was that web browsers parsed JSON so fast that parsing Transit-JSON to a js/Object and then translating the Object to data structures with Transit was faster than JavaScript could parse EDN.

This was a good overall glance on "edn as json" imo https://nitor.com/fi/artikkelit/pitfalls-and-bumps-clojures-extensible-data-notation-edn

EDN shines for human written data, not so much for machine written data. I would look into Transit for an API format.

> EDN shines for human written data, not so much for machine written data. I don't know that I agree with this. There's nothing that makes EDN worse than JSON for machines to read or write. EDN is often actually slightly smaller than the corresponding JSON (no commas, one fewer character for every key if using keyword keys rather than string keys (`:` vs ""), etc. It's just that JSON parsers tend to be insanely optimized, whereas EDN parsers don't (apart from https://github.com/tonsky/fast-edn/, I think).

Except that yeah, there's no clojure.edn/write, which is a shame.

generating (re-)readable EDN programatically is far from easy. Printing has to many side effects.

? Isn't jsonista a JVM library?

From Jsonista github-page > • Uses https://github.com/FasterXML/jackson-databind > • Mostly written in Java for speed But it is very good and fast json-library for Clojure.

We have this error now and then when the Cognitect test-runner fails with a strange error.

Running tests in #{"test"}
Execution error at cognitect.test-runner/restore-vars! (test_runner.clj:50).
No namespace: manta.action-handlers-test found
make: *** [Makefile:114: test] Error 1

The namespace differs it is quite random, and it does exist. A simple-rerun usually solves the problem, so I have a hunch that this is a race condition or something simiar.

Did anybody experience anything like that?

how are you running the tests? i see Make there

test:
	rep '(rrun "cognitect.test-runner.api/test" {})'

It is a project using deps.edn and we are using:

io.github.cognitect-labs/test-runner {:git/tag "v0.5.1" :git/sha "dfb30dd"}

What's rep? The only thing I can find is Sawfish, and surely it's not that. And what's rrun?

That is a convenient function of ours:

(defn rrun
  [var-str & args]
  (apply (rr var-str) args))

and rr is:

(defn rr
  [var-str]
  (requiring-resolve (symbol var-str)))

Still not clear what rep is. But, approaching it from the other side - does it ever fail if you run the tests manually, the way the test runner documents the process?

rep is this: https://github.com/eraserhd/rep

Our setup is quite complicated, because we spin up the same Clojure process to run the unit test that we use later for behavior tests and browser tests as well

Probably not related. One potentially pertinent fact - the No namespace: manta.action-handlers-test found error comes from calling (ns-publics ...). And it throws that error not when the namespace could not be loaded because none of the relevant files exist, but when the namespace was not required. So maybe you have some conditional requires. Maybe some dynamic requires. Maybe via some library like e.g. Integrant.

We are also using integrant, yes

> Our setup is quite complicated, because we spin up the same Clojure process to run the unit test that we use later for behavior tests and browser tests as well Nothing described makes it sound complicated. But it does smell a bit, to be honest. Could be that some tests implicitly depend on other tests. And you get that race condition because e.g. in 99% of the cases some namespace gets loaded in parallel testing but in the remaining 1% the loading order ends up being different.

That error comes from a particular test or a set of tests, probably. Find which one, and you'll probably find the culprit.

The problem is that the error message does not help since it hides the root cause

> Nothing described makes it sound complicated. I did not describe the whole thing, just the parts felt relevant to the problem

There should be a full stack trace somewhere. If there isn't, your setup is hiding it - not Clojure itself. Clojure's built-in REPL might redirect it to a file, but it also says so when it does that.

I cannot find any stack trace in the main process either

I am not blaming Clojure here, but I suspect that the test-runner has some strange issue

the test runner does some odd stuff. it gathers all test vars, updates the metadata of the vars to not have the tests anymore, runs the gathered tests, and then resets the metadata back to its original state

That might be related to the problem we can see here

requiring-resolve is not entirely thread-safe, and ISTR test-runner does dynamic require as well (as your own code), so this def. sounds like a race condition around those dynamic requires...

And rep is connecting to an existing, running REPL, and executing code in that context so that has existing state -- which is likely also part of this problem.

(see https://ask.clojure.org/index.php/12346/potential-race-condition-in-requiring-resolve for more deets)