I think that the WIP JS client isn't on rest API, it's on "peer API"

I think that just in @U050B88UR desktop 😕

I wonder what the best way to accomplish this would be. I tried a couple of things and have a working solution, but this requires a surprising amount of manual work. Feels like I a missing something to get a nice, idiomatic solution. The best I came up with, was using the query API but I ran into issues with optional refs.

@U066HBF6J look how close yours is to the index format: [e a v t]. you have {e {a v, a v]}. Perhaps you could group-by a d/datoms on :e, and then process the groups to make an a-v map?

you’ll have some edge cases around things like enums, but should be pretty straightfoward

Thanks, let’s see if I understood you correctly. You suggest to use d/datoms to retrieve datoms from the index by entity-id directly and then reshape that into the target data structure, right? I am already playing around with this but I struggle to see how to work with the datoms. Is there any documentation you could point to?

How do you determine the "relevant entities" (the keys in your map?)

I have a list of entity ids. For each entity from that list, I want to retrieve all attributes and values. Any ref attribute I would like to have as an entity id in the entity hash-map, but also as a corresponding top level entity (thus normalized data).

how many levels of recursion are you planning to go?

So, “relevant entities” are identified by the list of entity ids and all refs on those entities.

one level is sufficient

ah sry, two. however, a solution that could generalize on the recursion depth would be good in any case.

the recursion is what changes this from a simple massage of d/datoms or d/pull-many results.

(->> (d/q '[:find ?e2 ?a2 ?v
            :in $ [?e1 ...]
            :where
            [?ref-type :db/ident :db.value/ref]
            [?e1 ?a1 ?e2]
            [?e2 :db/valueType ?ref-type]
            [?e2 ?a2 ?v]
            [?a2 :db/valueType ?ref-type]]
       db entity-list)
     (group-by first)
     (into {}
       (fn [[e tuples]]
         [e
          (reduce
            (fn [acc [_ a v]]
              (let [{:keys [ident cardinality] attr-info} (d/attribute db a)]
                (case cardinality
                  :db.cardinality/one
                  (assoc acc ident v)
                  :db.cardinality/many
                  (update acc ident (fnil conj []) v))))
            {}
            tuples)])))

This will get the refs and collect them into scalar ids. To get all attributes from the first level, I suggest making a map from (d/pull-many db entity-list) and merging the results of this code into it to overwrite the ref-typed attributes

to generalize the query to multiple levels, you'll need a rule

what favila said 👏

the rule would probably look something like this

[[(follow-refs [?depth ?e1] ?a1 ?e2)
          [(> ?depth 1)]
          [?e1 ?a1 ?e2]
          [?e2]                                             ; ensures ref type
          [(dec ?depth) ?ndepth]
          [(follow-refs ?ndepth ?e2 ?a2 ?e3)]]
         [(follow-refs [?depth ?e1] ?a1 ?e2)
          [(= ?depth 1)]
          [?e1 ?a1 ?e2]
          [?e2]]]

(untested)

Thanks, I really appreciate providing that code and I will test it. Still surprised how much code is needed for that TBH.

It might be worth describing, WHY I want to have that data structure: I would like to export entities (incl. all entities associated via refs) from a Datomic instance A and import it to B. IDs are not being shared between the two DB instances. I thought having a normalised map of all the relevant entities would be the easiest for import (using the IDs as tempids). But maybe there would be a better way to achieve that export/import cycle? (Export should be handled by an application, exposing it via an HTTP API, so I don’t want to copy data on the storage layer for example)

I think exporting an [e a v] list is far simpler. then, when transacting, you only need to do the id -> tempid conversion, and datom -> tx assertion conversion (i.e. add :db/add at the beginning of the datom). which isn’t much code at all

This may interest you: https://gist.github.com/favila/785070fc35afb71d46c9 It's a little old (before mem storage had a working log, before string tempids, etc) but it demonstrates "application-level" datomic db dump and restore. There may be some ideas to mine.