Fork me on GitHub
#datomic
<
2018-01-02
>
devurandom08:01:30

David Nolen recently mentioned Datomic on the client (i.e. web browser) and from e.g. NodeJS [1]. However, the only Datomic client libraries for other languages like JS [2] appear ancient and unmaintained (last commits in 2015 for Ruby, Python, JS) and targetting the REST API only, which is considered "legacy and will no longer be supported" [3]. So what was David Nolen talking about [1]? [1]: https://www.youtube.com/watch?v=nbMMywfBXic [2]: http://docs.datomic.com/languages.html [3]: http://docs.datomic.com/rest.html

souenzzo11:01:05

I think that the WIP JS client isn't on rest API, it's on "peer API"

devurandom04:01:39

Is that WIP JS client already available somewhere?

souenzzo12:01:54

I think that just in @U050B88UR desktop 😕

nblumoe10:01:42

Hey, I would like to get a normalised map of entities from Datomic. So instead of having a map with nested entities, I would like to have all relevant entities on the top level and ids for entity links:

{12345 {:other/entity 54321 :this/name "foo"} 
 54321 {:this/name "bar"}}

nblumoe10:01:38

I wonder what the best way to accomplish this would be. I tried a couple of things and have a working solution, but this requires a surprising amount of manual work. Feels like I a missing something to get a nice, idiomatic solution. The best I came up with, was using the query API but I ran into issues with optional refs.

robert-stuttaford13:01:14

@U066HBF6J look how close yours is to the index format: [e a v t]. you have {e {a v, a v]}. Perhaps you could group-by a d/datoms on :e, and then process the groups to make an a-v map?

robert-stuttaford13:01:44

you’ll have some edge cases around things like enums, but should be pretty straightfoward

nblumoe15:01:01

Thanks, let’s see if I understood you correctly. You suggest to use d/datoms to retrieve datoms from the index by entity-id directly and then reshape that into the target data structure, right? I am already playing around with this but I struggle to see how to work with the datoms. Is there any documentation you could point to?

favila15:01:51

How do you determine the "relevant entities" (the keys in your map?)

nblumoe15:01:11

I have a list of entity ids. For each entity from that list, I want to retrieve all attributes and values. Any ref attribute I would like to have as an entity id in the entity hash-map, but also as a corresponding top level entity (thus normalized data).

favila15:01:53

how many levels of recursion are you planning to go?

nblumoe15:01:03

So, “relevant entities” are identified by the list of entity ids and all refs on those entities.

nblumoe15:01:15

one level is sufficient

nblumoe15:01:43

ah sry, two. however, a solution that could generalize on the recursion depth would be good in any case.

favila15:01:36

the recursion is what changes this from a simple massage of d/datoms or d/pull-many results.

favila16:01:41

(->> (d/q '[:find ?e2 ?a2 ?v
            :in $ [?e1 ...]
            :where
            [?ref-type :db/ident :db.value/ref]
            [?e1 ?a1 ?e2]
            [?e2 :db/valueType ?ref-type]
            [?e2 ?a2 ?v]
            [?a2 :db/valueType ?ref-type]]
       db entity-list)
     (group-by first)
     (into {}
       (fn [[e tuples]]
         [e
          (reduce
            (fn [acc [_ a v]]
              (let [{:keys [ident cardinality] attr-info} (d/attribute db a)]
                (case cardinality
                  :db.cardinality/one
                  (assoc acc ident v)
                  :db.cardinality/many
                  (update acc ident (fnil conj []) v))))
            {}
            tuples)])))

favila16:01:13

This will get the refs and collect them into scalar ids. To get all attributes from the first level, I suggest making a map from (d/pull-many db entity-list) and merging the results of this code into it to overwrite the ref-typed attributes

favila16:01:00

to generalize the query to multiple levels, you'll need a rule

robert-stuttaford16:01:22

what favila said 👏

favila16:01:51

the rule would probably look something like this

favila16:01:53

[[(follow-refs [?depth ?e1] ?a1 ?e2)
          [(> ?depth 1)]
          [?e1 ?a1 ?e2]
          [?e2]                                             ; ensures ref type
          [(dec ?depth) ?ndepth]
          [(follow-refs ?ndepth ?e2 ?a2 ?e3)]]
         [(follow-refs [?depth ?e1] ?a1 ?e2)
          [(= ?depth 1)]
          [?e1 ?a1 ?e2]
          [?e2]]]

favila16:01:58

(untested)

nblumoe08:01:08

Thanks, I really appreciate providing that code and I will test it. Still surprised how much code is needed for that TBH.

nblumoe08:01:07

It might be worth describing, WHY I want to have that data structure: I would like to export entities (incl. all entities associated via refs) from a Datomic instance A and import it to B. IDs are not being shared between the two DB instances. I thought having a normalised map of all the relevant entities would be the easiest for import (using the IDs as tempids). But maybe there would be a better way to achieve that export/import cycle? (Export should be handled by an application, exposing it via an HTTP API, so I don’t want to copy data on the storage layer for example)

robert-stuttaford12:01:58

I think exporting an [e a v] list is far simpler. then, when transacting, you only need to do the id -> tempid conversion, and datom -> tx assertion conversion (i.e. add :db/add at the beginning of the datom). which isn’t much code at all

favila14:01:48

This may interest you: https://gist.github.com/favila/785070fc35afb71d46c9 It's a little old (before mem storage had a working log, before string tempids, etc) but it demonstrates "application-level" datomic db dump and restore. There may be some ideas to mine.