Clojurians Log v2
datomic 2026-10-10
Some five-o'clock-on-a-Friday thoughts:
For years almost all of my Datomic tx data has been in the vector form, not the map form. I've also for years believed that it makes sense to have parent/child refs point in the upward direction (e.g. :node/parent with cardinality one vs :node/children with cardinality many), for the reasons Francis outlined here: https://favila.github.io/2023-04-18/datomic-choosing-ref-direction/.
But lately I've been working on a greenfield project dealing with very tree-shaped data and it seems like there are real advantages to being able to just build up nested Clojure maps in the natural way, and then transact them. But you can only do this if the references go in the downward direction:
{:node/id 1
:node/children [{:node/id 2}
{:node/id 3
:node/children [{:node/id 4}
{:node/id 5}
{:node/id 6}]}]}d/pull --in my example of above, :node/_parent instead of :node/children) into a flattened sequence of maps with the upward refs and the correct tempids.
Is there a reason this is a bad idea? The fact that Datomic does not natively support this gives me pause.
I think this is a fine idea. I don't think Datomic should support it natively because it's too opinionated. I think tx map syntax was already a bad idea and causes more confusion than convenience—eg thinking asserting a field in a map replaces card-many fields (datomic is not a document store), or applying specs to both the tx map and the pull result. Exposing only the primitives and writing your own map sugar on top forces you to reckon with the map/datom impedance mismatch, and see that there are different reasonable choices you could made about how map syntax maps to adds/retracts
We heavily use the map syntax (but not exclusively). Once you internalize a few design decisions (that asserting a map field augments a card-many attr 🙂 ) the benefits of map syntax shine. As you note, @enn, the natural match of map form to abstract entities and the applicability of core Clojure tools are really useful. Our devs find it very natural to think, design and debug in entity (map) form both when querying (pull) and transacting. On the transacting side, entity map form frequently needs to bottom out with an assertion that a referenced entity exists (e.g. an address that references a state) -lookup refs are perfect for this and we have a pre-processor (pxx) that converts referenced entities into lookup refs during a "dehydrate" phase. The end result is that our domain model is expressed in terms entity graphs: construction, db querying, db transacting, transmitting, displaying, etc. and Datomic does a good job as a domain entity graph store. The bits it doesn't address (richer types, reference assertions, etc) we smooth over with pxx. I have written tools that pre-process the map form into vector of primitives form -it's not hard but it does require a bit of "local schema" to recognize various subtleties like card-many references versus card-many "regular" attributes and tuples and such. I wish Datomic exposed that tooling to (cloud, ion) clients. If Datomic did not expose map form on transact, I would probably extend my existing tooling to do it -but I'm pretty happy with the design decisions in Datomic and I'm happy to hand that over to Nu. Same on the reading side, but there I think the value of pull is pretty settled. I have proposed a talk on pxx for the upcoming Datomic conference in Durham. My intent is to extract pxx to an open-source lib and if the talk is accepted that will motivate me quickly.
Oh, interesting, I'm hoping to be at that conference, so I'll look forward to it. Francis--thanks for the gut check. I was thinking "oh, if I do this then I can have functions don't care if their inputs are uncommitted or the result of a pull." Your point about the fundamental mismatches between maps and datoms is a good reminder that there are some traps there.
When I do an ion push, I get a large number of warnings, too many for the console buffer to even scroll up and see the first of them.
[main] WARN com.amazonaws.internal.InstanceMetadataServiceResourceFetcher - Fail to retrieve token
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
...
Caused by: java.net.SocketException: Host is down
...
... 77 more
What version of AWS CLI and what version of Datomic? Most importantly how are you passing credentials?
I am really not sure beyond that your AWS credentials provider chain is involved and its not able to reach ec2 so is throwing a warning. If you are running only locally, maybe this is just noise... if you ultimately connect its because it just keeps moving to the next provider in your /credentials and maybe you are getting 1 log per tried credential?
Thanks for your response and for some things to try. Versions of things:
$ aws --version
aws-cli/2.34.47 Python/3.14.4 Darwin/25.6.0 exe/arm64~/.aws/credentials , where I have a profile pointing to an IAM user specifically set up for these ion pushes and other datomic related activities. My call to ion/push explicitly points to it with the :creds-profile parameter, and none of the other profiles in the credentials file would work at all, they don't have the right policies/permissions.
It seems strange to me that AWS would move along and try profiles one after the other, that would be unobvious and possibly disastrous behavior. Besides, always in the past, when my profiles, IAM users, and policies are not exactly lined up, AWS errors out. Let's just say I have lots of experience with that. Also, I only have 6 or 7 (hah!) profiles in there, but there are over 70 warnings.
Well, it is curious and I'd like to clean it up, but at the end of the day it is not preventing me from deploying my tx-fn's.