Since this question and thread are inappropriate for #clojure and the thread got started in two other channels as well, I'm deleting all of this here and in #remote-jobs where it is also inappropriate. /admin

Sounds like you're trying to make an apple taste like an orange. If you have a specific need for one of those, just use it. If you can get away with creating a new PG type, go with that option.

Note also that conceptually, Transit is not for storage. And while you can disable its keyword caching, making it somewhat human readable, you still wouldn't be able to reasonably use PG's JSON querying facilities. Because Transit is not JSON, even though it wraps it.

And to answer your final question about the practice: 1. Come up with a list of requirements, where each requirement has a weight 2. Come up with a list of possibilities that I'd be willing to use 3. See which possibility fits the list of requirements the best There are many options, not just JSON or Transit.

Transit has a json encoding that will preserve types

> Transit has a json encoding that will preserve types It becomes impossible to instruct PG to select json fields and other stuff.

Conceptually Transit is just fine for storage, not sure why you'd say that

Json is fundamentally lacking in the ability to preserve information

If your constraint is that you need json for querying in pg then the question is how you work around that

> Conceptually Transit is just fine for storage, not sure why you'd say that Right, I should've added that doing that imposes extra constraints on the developer that the developer must be aware about.

@U2FRKM4TW I’d like something that is possible to convert clj map where values are keywords into a json, and then 100% convert it back to clj maps. Also, in the meantime, the PG Json support can work on the serialized data. Sounds impossible to achieve though.

If all your keys are keywords, then just convert string keys to keywords when reading the JSON from PG. If not all your keys are keywords, you have other options: • Invent your own encoding • Invent your own type • Add a spec/schema to the data and use it to keywordize the right keys

> • Add a spec/schema to the data and use it to keywordize the right keys Well. I was also thinking on the ability of coercing values with malli. I will have a check.

The problem I see here is that you will have a bit of a mismatch between how you use EDN in your application code vs the serialised version in PG. If you want to leverage PGs json capabilities then you probably want to use a subset of EDN, coerced to JSON and back with specific rules. Meaning you want to make a whole bunch of assumptions about how your EDN will look so you don't get lost in generality and can leverage PG more easily. clojure.data.json can do that for you. write/write-str and read/read-str have arguments where you can pass in functions for both key and value translation.

Even with malli, you'll still suffer from slight impedance mismatch if you query the json from pg as it doesn't have any notion of keywords

Wouldn't Transit with JSON-Verbose work?

The biggest limitation of JSON to get around is that keys must be strings. There's no easy way around that. Where for values you can just wrap them in a map with a "type" and "value" key that describes the real type, to do the same for the keys in JSON you need a string encoding. That's what transit does basically. And it also handles all the edge cases, like how there's a limit to how big a number can be in JSON, so it switches to a string with a special encoding again if you try to serialize a bigger number for example.

But, since you want to use PG's support for JSON, my guess is you'd want to coerce all types to some JSON type instead, and when you read it back, I'd coerce it back based on some known schema. You can't make it self-describing in JSON inside PG without breaking PG's abilities to query it, since it can't query custom types and those don't even exist in the JSON standard.

This will have limits similar to static types. For example, you can't have keys of different types since you won't know what to coerce those back into if that's the case.

transit makes things unreadable.

Even if you use :json-verbose?

Transit is too locked in. I would go with json + malli. and nippy for compression if needed.

Nippy is a binary serialization, how would that work with PG?

My guess is you don't really need perfect type preservation, if you do, then Transit is hard to beat. But probably you can get away with just stringifying keys and keywordizing them back. And generally making sets into JSON lists and such things, and probably you don't need to support numbers bigger than JSON does and all that. Basically, you need to relax your requirement of "it loses the information that originally it is a keyword and not a string.", if you allow for a lossy conversation from Clojure to JSON and back, you can just rely on JSON.

> Nippy is a binary serialization, how would that work with PG? Yup. Nippy is irrelevant here.

> if you allow for a lossy conversation from Clojure to JSON and back, you can just rely on JSON. Yes. That’s the direction I am going. Transit is designed, and as a consequence, restricted in Clojure space.

Transit isn't restricted to Clojure, it can apply generally to any language, see: https://github.com/cognitect/transit-format , but if you mean that it doesn't have a lot of support outside of Clojure, yes, you won't find very many serialization schemes that have broad reach beyond JSON and CSV, and maybe Protobuf

> Transit isn’t restricted to Clojure, it can apply generally to any language, see: This kind of description is often misleading to new comers. As an wire format, like you said, it does not gain much attraction outside of the Clojure world. Similar things could be core.async, it is even not preemptive but is marked as a selling point that could be compared with golang csp. (Though I still use it, better have than nothing)

I mean, nothing is false about the statement, what's the alternative, just not say anything about what it does?

There's an official implementation, for Java, Python, Ruby, JS, Clojure and Clojurescript of Transit, but ya, none of those outside Clojure are probably widely in-use. But then Clojure itself isn't very much widely used either. It's a chicken and egg though, Transit isn't going to see more adoption if everyone looking for a serialization scheme to adopt use "most commonly used" as their criteria.

Also, just an FYI, Go-lang did not support preemtive either back when core.async released, it used to be only cooperative. They added that in later versions.

Fair enough.

I understand where you're coming from though, but it just seems silly to add warnings like: "Warning, Clojure is a niche language, not many people use it, proceed at your own risk." when at the same time, you want more people to use it 😛 Also, for core.async, I recommend this reading: https://clojure.org/news/2013/06/28/clojure-clore-async-channels I think sometimes people get mislead but it's not from the official docs, I found the http://clojure.org content is always the best.

What's fun about this read, is it mentions that it's actually taken a similar approach to C#'s async, while providing CSP constructs. And that's a very accurate description. The way Go implements CSP is very different, it uses real fibers, aka, lightweight threads, but core.async uses a state machine like C#. Clojure is also stackless like C#, and unlike GO, which is stackful.

I understand your points and I did hope the community and ecosystem become better. To that end, attracting more users is a necessary step. Also, what I want to say is some wording might be more objective. Not specially about your part on transit, it’s just a general feeling about the articles in the community.

That's possible, it's hard to quality control what people are going to write about, but that's why I try to direct people to the official doc, http://clojure.org and http://ask.clojure.org, there's really good objective descriptions and explanations on those.

Iirc there's a company which uses clojure for robots

Presentation from Clojure Conj 2019 https://youtu.be/HXJaJTJe0Hg

Flying robots with Clojure https://youtu.be/Ty9QDqV-_Ak

Maybe this is of interest too https://www.petekeen.net/building-bots-with-clojure

thank you !!

I’ve been looking for a orgmode supporting generator personally (and depending on hugo which has native support) Don’t know how useful/helpful this is - but https://github.com/200ok-ch/org-parser seems like a useful attempt.

@U06KGH6LB Oh wow I didn't know about org-parser. This is exactly what I'm looking for because it has a good intermediate representation as well as typical html hiccup, just like cybermonday. The idea I have is letting the user create mapping functions at both levels. That's what I already have for markdown. I'm feeling inspired to add org support now.

If you ever open it up, would be eager to try! (Not sure if I’ll be capable - but also to contribute)

I'm deliberately avoiding boosting it too much because it's in alpha right now, but it is open! You can poke around at https://github.com/stelcodes/nuzzle. It's definitely in the "API breaking every weekend" stage right now but it's really coming along and I hope to give it a 1.0.0 tag by June. I would love issues and PRs but I will say there is a strong focus on simplicity and API design. I'm trying to keep things very obvious and consistent. The vision is basically a static site generator you can bolt onto an existing project because it's so lightweight.

@U06KGH6LB dang I actually spoke too soon about org-parser having hiccup support. It actually doesn't but I just opened an issue and asked if anyone knows about efforts related to making one. That is absolutely essential to including org-mode in Nuzzle so if you want to try to write that, that would be huge. It could be an amazing little library by itself or it could be included in Nuzzle or even org-parser if they want it (I just asked). https://github.com/200ok-ch/org-parser/issues/64

To answer the initial question, I don't see markdown only as huge problem as the vast majority of content is written that way. All the content for http://practical.li is written in markdown, although the books use a generator that adds a lot of extra features that markdown does not support Asciidoc support is more prevalent in technical documentation and is arguably a better format for generating richer output as its output can be customised far more I believe (for example I hope to use asciidoc with cryogen to generate html with CSS tags for a CSS framework, automatically providing a more engaging website experience) Org-mode is also an amazing format for generating a wide variety of output formats, with many converters and generators. Org is quite Emacs centric though, so unsure how easy it would be to integrate or how widespread. I assume there is a higher proportion of Clojurians using org mode than the general developer audience

@U05254DQM Hey thanks for the thoughtful response. I'm glad to see some Markdown love. I like how simple it is, but it's definitely not "super-user" powerful. And since my library is going to be most accessible to Clojure devs so I would love to support org-mode for all the emacs dwellers. And I would love to support Asciidoc as well but I haven't seen a library that can do Hiccup-esque intermediate representation and also transform to Hiccup. I'm realizing that's a requirement I want to have for all markup support in my SSG. That way there can be a uniform API and no "second-class" markup syntax. It would be kinda magical to have my code-block syntax highlighting (for example) work the same way on all supported markup types. That's where a good intermediate representation really comes in handy.