Clojurians Log v2

i've just realized that resolve-if-ref is not taking neither cardinality nor the isComponent flag into account yet, and im not sure how to handle them yet either. specifically the :db/isComponent true attrs, might need domain specific handling, because u might want to replace the set of values whole sale, atomically, which would require retractions too. here is some further reading: • https://forum.datomic.com/t/replace-cardinality-many-attribute/759/2 ◦ https://stackoverflow.com/questions/42112557/datomic-schema-for-a-to-many-relationship-with-a-reset-operation ▪︎ https://github.com/vvvvalvalval/datofu#resetting-to-many-relationships

this bothers me to no end too, so i just wrote some workarounds for it. i'll see, if i can extract it from our code base. it's a problem on Datomic Cloud too. there was a atomic forum thread about this question too where it was explained that it's not possible to solve this in a general way

u must use anon functions, if u want to use logic vars deeper than the top level of expression, because they are only substituted at the top level

> there was a atomic forum thread about this question too where it was explained that it’s not possible to solve this in a general way Ah, I see:pensive: I needed this confirmation:sweat_smile: Would be cool to have documentation section in https://docs.datomic.com/schema/schema-reference.html#tuples which describes all refs specifics:thinking_face:

we use this convenience function often on datomic cloud:

(defn eid
  "It returns the entity ID of `entity:ref` or `nil` if there are no facts
   associated to it.

   `entity:ref` can be one of the following:

   1. nil
   2. an entity ID
   3. a `:db/ident` keyword
   4. a lookup-ref

   It's useful in queries, where a query input is an ident, but it should
   participate in a data-pattern of a datalog `:where` clause, which has a
   dynamic attribute (eg. a logic var or `_`).

   It's also useful for checking the existence of an entity at the basis-t of
   `db-val`.

   It's similar to the Datomic Pro `d/entid` function:
   
   "
  [db-val entity:ref]
  (when entity:ref
    (-> db-val
        (d/datoms
          {:index      :eavt
           :components [entity:ref]
           :limit      1})
        first :e)))

(defn resolve-if-ref
  "Resolves the value of a Datomic entity map's map-entry, if its `attr` is a
   ref attr and returns the map-entry, with the resolved value."
  ([db-val* [attr* value*]] (resolve-if-ref db-val* attr* value*))
  ([db-val attr value]
   (if (-> attr (= :db/id))
     [attr value]

     (let [{:as _attr-def :keys [db/valueType db/tupleAttrs]}
           (d/pull db-val [[:db/valueType :xform `gdx/ident] :db/tupleAttrs] attr)]
       [attr (case valueType
               :db.type/tuple
               (mapv (comp second (partial resolve-if-ref db-val))
                     tupleAttrs value)

               :db.type/ref
               (eid db-val (if (vector? value) ; lookup-ref?
                             (resolve-if-ref db-val value)
                             value))

               value)]))))

i wrote this resove-if-ref primarily to help implementing a generic upsert transaction function, which would be capable of resolving tuple attrs. unfortunately, because of the above mentioned "tempid resolution within tx fn" issue, we couldn't make it completely generic:

(defn ^:datomic/tx upsert
  "Tuple-attribute key aware upsert.

   If there are subsets of keys of the `entity`, which can define a unique
   tuple attrs, assoc the 1st such tuple attr into `entity`, with its
   corresponding lookup-ref value.

   If the entity exists, dissoc the components attrs of that 1st tuple attr,
   so the entity will behave as an update, when transacted, instead of
   throwing a `:db.error/unique-conflict` error."
  [db-val entity]
  (let [resolved-entity (into {} (map (partial resolve-if-ref db-val)) entity)

        [{:as   first-key
          :keys [key/ident key/vals key/tuple-attrs]}]
        (tuple-lookup-refs db-val resolved-entity)]

    [(-> resolved-entity
         (cond-> (and (some? first-key)
                      (eid db-val [ident vals]))
                 (as-> e
                       (assoc e ident vals)
                       (apply dissoc e tuple-attrs))))])

where the tuple-lookup-refs fn is this:

(defn tuple-lookup-refs
  "Compute all possible tuple-attribute based lookup refs of `entity`.

   FIXME:
   It resolves ident values of `:db.type/ref` attrs in the tuple to their EIDs,
   but not sure why...

   Eg. if `:a` & `:b` are `:db.type/string`
   and `:a+b` is a `:db/tupleAttrs [:a :b]`:

     {:a \"X\" :b \"Y\"}
     =>
     [[:a+b [\"X\" \"Y\"]]

   Eg. if `:s` is a `:db.type/string` and `:r` is `:db.type/ref`
   and `:s+r` is a `:db/tupleAttrs [:s :r]`:

     {:s \"X\" :r :some-entity}
     =>
     [[:s+r [\"X\" <eid of :some-entity>]]
   "
  [db-val entity]
  (-> '{:find  [?tuple-attr-ident ?lookup-vals ?attrs-of-a-tuple-attr]
        :in    [$ ?entity-map]
        :keys  [:key/ident :key/vals :key/tuple-attrs]
        :where [[?tuple-attr :db/valueType :db.type/tuple]
                [(missing? $ ?tuple-attr :db/isComponent)]
                [?tuple-attr :db/ident ?tuple-attr-ident]
                [?tuple-attr :db/tupleAttrs ?attrs-of-a-tuple-attr]
                [((fn [tuple-attrs entity-map]
                    ;; ignore unrelated tuple attrs
                    (not-empty
                      (clojure.set/intersection (set (keys entity-map))
                                                (set tuple-attrs))))
                  ?attrs-of-a-tuple-attr ?entity-map)]
                [((fn [$ entity-map tuple-attrs]
                    (mapv (fn [a-tuple-attr]
                            ; resolve the value of a a-tuple-attr, if it's a ref
                            ; attribute, otherwise use its value from the entity-map
                            (when-let [lookup-val (get entity-map a-tuple-attr)]
                              (-> $
                                  (datomic.client.api/datoms
                                    {:index      :eavt
                                     :components [lookup-val]
                                     :limit      1})
                                  first :e
                                  (or lookup-val))))
                          tuple-attrs))
                  $ ?entity-map ?attrs-of-a-tuple-attr)
                 ?lookup-vals]]}
      (q db-val entity)))

these are the tests for this upsert function:

(deftest upsert-test
  (testing "single tuple id"
    (let [txs
          [;; Schema
           ;;
           ;; Use a symbol for the 2nd part of the composite key, because a
           ;; - :db.type/long looks like an EID
           ;; - :db.type/string looks like a tempid
           ;; - :db.type/keywords looks like an ident
           ;; - :db.type/uuid is too long
           ;; - :db.type/date is verbose to construct, might appear as a long too
           ;; - :db.type/tuple looks like a lookup ref
           [(dc/mk-attr :p1 :db.type/ref)
            (dc/mk-attr :p2 :db.type/symbol)
            (dc/mk-attr :k :db.type/tuple 1 'id :db/tupleAttrs [:p1 :p2])]

           ;; Example value for the :p1 ref attr
           [{:db/ident :x}]]

          attrs [:db/ident [:p1 :xform `gdx/ident] :p2 :k :db/doc]
          dc0 (-> dc/kit (assoc :txs txs) (rmap/valuate-keys! :txrs))]

      (testing "insert txd"
        (testing "when ref part is an existing ident"
          (let [dc (-> dc0 dc/speculative)
                e {:p1 :x :p2 'x :db/doc "insert"}]
            (is (match?
                  [(assoc e :p1 (dc/eid dc :x))]
                  (dc/upsert (dc/val dc) e)))))

        (testing "when ref part is an undefined ident"
          (let [dc (-> dc0 dc/speculative)
                e {:p1 :UNDEFINED :p2 'x :db/doc "insert"}]
            (is (match?
                  [(assoc e :p1 nil)]
                  (dc/upsert (dc/val dc) e))))))

      (testing "with updated txd"
        (testing "using ident as ref value"
          (let [dc (-> dc0 dc/speculative)
                e {:db/ident :e :p1 :x :p2 'x :db/doc "insert"}
                e' (assoc e :db/doc "UPDATE")]

            (->> [[`dc/upsert e]]
                 (dc/tx! dc))

            (is (match?
                  [{:p1     m/absent
                    :p2     m/absent
                    :k      [(dc/eid dc :x) 'x]
                    :db/doc "UPDATE"}]
                  (dc/upsert (dc/val dc) e')))

            (testing "when transacted"
              (is (match?
                    e'
                    (-> (dc/tx! dc [[`dc/upsert e']])
                        (dc/txr-pull attrs :e)))))))

        #_(testing "using tempid as ref value"
            (let [dc (-> dc0 dc/speculative)
                  tempid-for-refed {:db/id ":x-tempid" :db/ident :x}
                  e {:db/id "e" :p1 ":x-tempid" :p2 'x :db/doc "insert"}
                  e' (assoc e :db/doc "UPDATE")]

              (->> [tempid-for-refed
                    e]
                   (dc/tx! dc)
                   ?)

              (is (match?
                    [{:p1     m/absent
                      :p2     m/absent
                      :k      [":x-tempid" 'x]
                      :db/doc "UPDATE"}]
                    (dc/upsert (dc/val dc) e')))

              (testing "when transacted"
                (is (match?
                      (-> e' (dissoc :db/id))
                      (-> (dc/tx! dc [tempid-for-refed
                                      [`dc/upsert e']])
                          ? (dc/txr-pull attrs "e")))))))))))

Hah, that is a lot=))) Interesting, with integrated peer I never thought about declaring anonymous function in datomic query:thinking_face: Never thought about it as possibility:thinking_face: