Clojurians Log v2

It’s simpler, with the minor sacrifice that you have to make assumptions that aren’t necessarily universal, like reading ":some/string" as a keyword and ?datalog-var to a symbol. I’ve never ended up needing to add parsing logic around escape characters or anything for that, but I imagine someone in some niche somewhere might. 🙂 Apart from that we use the datalog json parser to do structurally aware parsing into symbols, functions, clauses, etc., w/spec collection regex encoding a subset of the datomic query grammar. Mostly you end up with similar affordances to edn, in that dicts and lists are python data structures with strong language support for literals and programmatic construction, etc. E.g., append a clause, change something in one clause at an index, put an argument into both the args list and the :in query clause list, etc. It’s more verbose per query to do that versus putting it in a raw string in Python, but then also, it’s just data then, instead of a string. The bigger thing you give up is not having all the smarts around segment caching and serialization, etc. you get in e.g. the peer library. Obviously, if you go to full semantic correspondence with edn and smarts around caching and serialization and compact encoding/decoding, and support for Python’s own data literals and collections for constructing things, you end up re-inventing transit. We didn’t end up using Transit though b/c we were also supporting R, but all the pain points we encounter are almost all the performance issues that transit solves, rather than needing richer edn semantics vs. heuristics around plain string encode/decode, so we’ve been considering just adding a content type request layer to support it.

Thanks for sharing! I checked your approach, you send JSON into Datomic? Would you consider it easier than using EDN? Asking because I'd like to stick with EDN and at the time you shared the links I had started writing actual code to write/query data and my initial idea was to use this https://pypi.org/project/edn-format/ however I have quickly discovered how verbose it can get trying to convert information into EDN so I went into validation-first direction where queries are written as templated EDN (via Jinja) then passed into edn.dumps(edn.loads(...)) to validate actual EDN before passing it forward to Datomic. I'd packaged that, the article I wrote, and some API nuances irrelevant to someone who just wants to get started with Datomic into this https://pypi.org/project/jinjatomic/

Hm I understand peers are a powerful way to essentially pull in (or distribute) a part of the dataset and leverage Datomic basically wherever. I also understand Datomic REST API is likely limited in that regard, plus it's guaranteed to remain limited, to quote docs, "no longer supported for new development". So if one wants to use these powerful features they're likely bound to implement somewhat of a query builder Python-side anyway?

I’ve supported a lot of Python & R use in the past (via a limited query-only API service), and I’ll say the nice thing about the LLM era is that LLMs have zero trouble writing datalog queries and don’t have developer experience issues 😆 so the “this is too weird” issues are vanishing. broken record in this world, but the dev experience thing is largely a matter of familiarity. True, there could be more supports out there for newbies, but anyone who has setup dependencies for a scientific python project w/packages w/native deps only in conda, and other pure python ones only in pip… then get that environment to function in production, or for reproducible analysis… Let’s just say they’ve got nothing to complain about in dev experience in comparison to just setting up your average Clojure or Datomic project. 🙂 They’re just already familiar with the typical pain points in their ecosystem.

Yep, yep, that's my starting point in a way. Someone wants to try out Datomic for their Python project but have never before come into significant contact with the Clojure or Java ecosystem and suddenly you're faced with a lot of things on top of Datomic itself. If the entire process could be cut down it'd be amazing! 😁

if you want to see some examples of the kind of use I’ve had to support: • query generation in plain python data structures -> plain json https://github.com/RCRF/patternq/blob/main/patternq/dataset.py • data driven import of tables into datomic https://unifybio.github.io/import-config/ while that’s all bio/translational data science specific, the core tooling is agnostic to any particular datomic schema and driven by additional attribute annotations and metadata stored in Datomic (via https://github.com/vendekagon-labs/unify) There can be a pretty big and near-instant impact for a team by getting a bunch of stuff distributed in ad hoc tables left lying around by people working in R and Python into one Datomic instance, available for general query by a wider team.