datomic 2020-08-04 | Slack Archive

@jaret @marshall what does it mean if i can see a datom in a d/db but not in a d/history of that same db?

@robert-stuttaford any chance the attribute has :db/noHistory set to true?

@robert-stuttaford second thought, you’re getting the history db from the db you see the datom in? If so, that sounds like something we would want to investigate. Would you be able to give us a small repro or better yet, a backup that shows this behavior?

robert-stuttaford13:08:40

that's right - db and (d/history db)

robert-stuttaford13:08:36

@jaret it's in our prod db, which has all our PII in it, started circa 2012 🙂

robert-stuttaford13:08:55

perhaps we could arrange a zoom and i could show you via screen share, and then we can see about next steps from there?

Lennart Buit14:08:05

Hey there, we are using datomic and are currently diagnosing a performance issue related to a recursive rule. We have a tree structure in datomic, that for each node, links a parent (or not), so a schema like this:

(def schema
  [;; Additional tree attributes omitted
   {:db/ident       :node/parent
    :db/valueType   :db.type/ref
    :db/cardinality :db.cardinality/one}])

Now, we sometimes look through the tree to find, say a descendant of a node. To do so, we have a recursive descendants-of? rule that binds this collection of descendants to ?node:

(def descendants-of?
  '[[(descendants-of? ?root ?node)
     [?node :node/parent ?root]]
    [(descendants-of? ?root ?node)
     [?node :node/parent ?intermediate]
     (descendants-of? ?root ?intermediate)]])

So far so good, we can do queries that reason about descendants, for example finding all descendants for a root:

(d/q '[:find ?node
             :in $ ?e %
             :where
             (descendants-of? ?e ?node)]
           (d/db (conn))
           root-eid
           descendants-of?])

Now, sometimes we have a candidate set of nodes, and of those candidates, we need to find the descendants, say like this:

(d/q '[:find ?node
             :in $ ?name %
             :where
             ;; assuming that `?name` only exists in one tree
             [?e :node/name ?name]
             (descendants-of? ?e ?node)]
           (d/db (conn))
           “name”
           descendants-of?])

In the most pathological case, where all nodes are named ?name, we will be binding all nodes in a tree to ?e, and then find the descendants of those ?e s and bind those to ?node. The result will be the same as the query above: all nodes but the root. However, this query appears to be much slower. I think that makes sense intuitively if we assume that descendants-of? is kinda expanded per ?e. For each member of ?e, we can potentially redo descendant seeking, if those descendants are also in ?e. Is there a way to optimise here, if there are potential queries that bind descendants and ancestors to ?e ?

➕ 3

favila18:08:26

Why not:

'[[(descendants-of? ?root ?node)
   [?node :node/parent ?root]]
  [(descendants-of? ?root ?node)
   [?intermediate :node/parent ?root]
   (descendants-of? ?intermediate ?node)]]

favila18:08:13

The second implementation of decendents-of? necessarily scans all :node/parent if ?node is unbound

favila18:08:54

in fact, datomic has a syntax for ensuring a rule input var is bound:

👀 3

favila18:08:07

'[[(descendants-of? [?root] ?node)
   [?node :node/parent ?root]]
  [(descendants-of? [?root] ?node)
   [?intermediate :node/parent ?root]
   (descendants-of? ?intermediate ?node)]]

Lennart Buit18:08:20

Ah let me take a look, I may have simplified the example a bit too much when lifting it from our codebase

Lennart Buit19:08:04

Oh you are on to something! Thank you so much

kenny18:08:15

Using :as with :db/id does not appear to have any effect. Is this expected?

(d/q
    '[:find (pull ?e [(:db/id :as "foo")])
      :where
      [?e :db/ident :db/cardinality]]
    (d/db conn))
=> [[#:db{:id 41, :ident :db/cardinality}]]

kenny19:08:05

Opened a support ticket for this https://support.cognitect.com/hc/en-us/requests/2797

kenny18:08:36

Other db/* attrs work:

(d/pull (d/db conn)
          '[(:db/id :as "foo")
            (:db/ident :as "ident")]
          :db/cardinality)
=> {:db/id 41, :db/ident :db/cardinality, "ident" :db/cardinality}

kenny18:08:23

This one is strange since :db/ident is included twice.

kenny18:08:43

:db/doc is not included twice.

(d/pull (d/db conn)
          '[(:db/id :as "foo")
            (:db/ident :as "ident")
            (:db/doc :as "doc")]
          :db/cardinality)
=>
{:db/id 41,
 :db/ident :db/cardinality,
 "ident" :db/cardinality,
 "doc" "Property of an attribute. Two possible values: :db.cardinality/one for single-valued attributes, and :db.cardinality/many for many-valued attributes. Defaults to :db.cardinality/one."}

souenzzo18:08:31

@kenny :db/id isn't an attribute. where is no Datom[e :db/id v]. It's a "special thing" that d/pull assoc into it's response.

kenny18:08:52

So? I don't think pull requires the selection to be attributes. From the doc "Pull is a declarative way to make hierarchical (and possibly nested) selections of information about entities."

kenny18:08:06

Even if that was a requirement, I don't think it makes sense for it to behave differently than everything else.

souenzzo18:08:52

I agree that it's a bug

2020-08-04

Channels