Fork me on GitHub
Ben Sless07:11:42

I noticed that medatada is not part of the EDN specification, only reader tags, yet the EDN read functions do keep metadata. Is metadata considered part of the formal spec?


I don't know any details but I think there's a trouble with languages that don't support metadata. Here's an old issue mentioning the same thing:


read-string and clojure.edn/read-string both support a superset of the edn specification clojure itself is a superset of edn for example, you can parse (clojure.edn/read-string "1/2") and it is not part of edn specification. I think that the idea of the edn specification is to keep it minimal, to be easily portable, and the ones that want, can implement it own extensions. for example, cljs do not support Ratios, if Ratios were part of edn specification, the cljs version of edn/read-string would be incomplete. So do not have ratios in edn spec were a good thing, because allow a full implementation in cljs

Alex Miller (Clojure team)12:11:18

Many languages do not have any way to implement metadata so it would be difficult for that to be portable

Ben Sless14:11:25

If I were to do something perverse and create a EDN serialization codec in protobuf, should I support only the official spec or fully support metadata? What about things like ratios?

Alex Miller (Clojure team)14:11:01

what problem are you trying to solve?

Ben Sless15:11:13

There you go asking the architect-y questions. Mostly just fooling around to see how easy it is to implement and compare the performance against other options. But if it's viable and people actually want to use it, I might as well do it properly, no?

Alex Miller (Clojure team)15:11:56

well I don't think you can answer the question above until you know what people might use it for

Alex Miller (Clojure team)15:11:15

metadata, ratios, etc are not edn, they are extensions to edn

Alex Miller (Clojure team)15:11:14

I don't get why anyone would want this

Alex Miller (Clojure team)15:11:43

like what does it get you that transit doesn't?

Ben Sless15:11:55

Don't get me wrong, I find it difficult to put into words how much I dislike protobuf

Ben Sless15:11:35

But if for some reason the performance and binary size are more important to users than things like ergonomics, visibility and sanity :man-shrugging:

Alex Miller (Clojure team)15:11:11

would it be competitive with nippy?

Alex Miller (Clojure team)15:11:19

is that b/c of fixed record knowledge?

Alex Miller (Clojure team)15:11:56

I'm just trying to figure out in which cell of the multi-dimensional matrix this is interesting

Ben Sless15:11:23

not sure I follow re: fixed record knowledge

Alex Miller (Clojure team)15:11:05

what is the question where this is the best answer?

Ben Sless15:11:27

Probably the cases where you'd want to use nippy

Alex Miller (Clojure team)15:11:33

cross language + fixed schema + ???

Alex Miller (Clojure team)15:11:46

nippy isn't fixed schema though is it?

Ben Sless15:11:31

It doesn't just serialize everything you give it. It has a pretty wide range of support, but if you throw some ad-hoc object at it it will break

Alex Miller (Clojure team)15:11:45

sure, but you don't have to know it up-front - it's serialization

Alex Miller (Clojure team)15:11:57

doesn't protobuff require some prior schema knowledge?

Alex Miller (Clojure team)15:11:10

or am I just out of touch with protobuf

Ben Sless15:11:37

It does, but what if the schema was the set of types supported by EDN? 🙂

Alex Miller (Clojure team)15:11:34

that seems ... uninteresting

Alex Miller (Clojure team)15:11:47

the schema+perf is the interesting difference

Ben Sless15:11:02

yes, well, not all experiments yield interesting results

Alex Miller (Clojure team)15:11:11

if this is just mapping Clojure types to protobuf, I mean, that's been done

Ben Sless15:11:26

Has it? Haven't come across it

Alex Miller (Clojure team)15:11:16

I think if you just google clojure and protobuf, you'll find things

Alex Miller (Clojure team)15:11:46

Ghadi was working on a pretty involved protobuf thing for a while but I don't know that he ever got around to releasing it

Ben Sless15:11:59

There's this library developed by a colleague of mine

Alex Miller (Clojure team)15:11:10

I think I did one like 10 yrs ago when I was comparing to avro, but again was never released

Ben Sless15:11:11

Gives a map-like facade over proto messages

Alex Miller (Clojure team)15:11:45

every time I've tried I always decided I hated proto and/or fixed schema so much that I did something else :)

Ben Sless15:11:57

Smart choice 😄

Ben Sless15:11:10

Protobuf - a solution in search of a problem

Ben Sless15:11:06

regarding formats, how do transit and fressian hold up? Is there a particular reason to prefer fressian?

Alex Miller (Clojure team)15:11:28

I think Stu's conj talk about Fressian lays out the benefits pretty well

Ben Sless15:11:53

I'll check it out 👍

Alex Miller (Clojure team)15:11:23

Datomic uses Fressian for some of its storage

Alex Miller (Clojure team)15:11:46

I think the biggest fork in the road is: do I have some out of band way to tell you about the structure of this data or is that encoded in the data itself? edn, transit, and fressian are all encoding schemes that are both extensible and include the structure in the data. protobuf is not afaik.

Alex Miller (Clojure team)15:11:36

so that's why I say that that is the interesting difference if it allows you to also achieve much better performance


I’ve used protobuf… in retrospect, it’s way too much hassle for what you get; It’s not a great fit for the dynamic + flexible nature of Clojure

Ben Sless07:11:23

It's not a great fit for any flexible system

yes 1