Fork me on GitHub
#xtdb
<
2023-08-02
>
Jenny Kwan17:08:01

How do I enforce uniqueness on an attribute that isn't the document ID? In order to enforce this properly, the read to check uniqueness has to happen in the same transaction scope as the write, and it looks like match is the only way to read in a write transaction scope. But match requires knowing the document ID of the conflicting document upfront? How does this work?

malcolmsparks17:08:57

Hi Jenny. I think you need to create a transaction function.

malcolmsparks17:08:35

You will be able to read the database to check that no documents exist with a conflicting attribute value.

Jenny Kwan17:08:36

The alternative, using a transaction function, would be fine, but I'm having trouble understanding how a Clojure JVM instance would bootstrap and load the transaction functions at startup. I happen to be using Mount. Would I just add a hook to mount/start to load all transaction functions? I'm paranoid about append-only versioning of code; when loading a transaction function, how can I ensure that it's either created new or is unchanged? Are there best practices for transaction function versioning?

refset19:08:20

Hey @me1408, the safest & least confusing thing to do is to simply avoid re-using names between changes. Stick a number on the end e.g. my-tx-fn2. Submitting identical versions of functions idempotently during the start of your application should be sufficient without any meaningful downsides (asides from a ~few superfluous entries on the tx-log). Malcolm's suggestion of using SCI could be interesting & appropriate, but if you were looking to keep the number of components to a minimum I would recommend the above.

Jenny Kwan00:08:40

That makes sense. SCI seemed like too much for this case. My only remaining question on this is how to detect whether the function being submitted is indeed identical. Will ::xt/match work on function bodies?

refset08:08:50

> Will ::xt/match work on function bodies? 100% yep, https://www.youtube.com/watch?v=oGkCwSdZBds 🙂

Jenny Kwan17:08:57

Another way of asking this is what are best practices for using temporal versioning of transaction functions? Since the version of the transaction function being called in a write transaction will necessarily match the document being written, the logging is already there. Are there patterns that already leverage this?

Jenny Kwan17:08:24

Just think typing: given the reality of rolling restarts for version migrations of the Clojure client, would it make sense to add the Clojure client code version to the name of the transaction function to ensure the function being called actually corresponds to the client code calling it? I guess a more paranoid approach is actually maintaining a node ID, let's say on the Docker container, and using that node ID in the transaction function name. There aren't that many nodes relative to data size.

hairfire17:08:19

I was playing around with a tiny subset of the data from the "Learn XTDB Datalog Today" page (see https://nextjournal.com/try/learn-xtdb-datalog-today/learn-xtdb-datalog-today) trying to convert from XTDB 1.x to XTDB 2 EA. Here's the XTDB 1.x code:

(ns rlws.xtdb-datalog
  (:require [xtdb.api :as xt])
  (:gen-class))

(comment

  (def my-node (xt/start-node {}))

  (def my-docs
    [{:person/name "George Miller", :xt/id -142}
     {:person/name "George Ogilvie", :xt/id -147}
     {:movie/title "Mad Max Beyond Thunderdome", :movie/director [-142 -147], :xt/id -218}])

  (xt/submit-tx my-node (for [doc my-docs]
                          [:xtdb.api/put doc]))
  (xt/sync my-node)

  (defn q [query & args]
    (apply xt/q (xt/db my-node) query args))

  (q '{:find [title]
       :where [[_ :movie/title title]]})

  (q '{:find [director-eid]
       :where [[movie-eid :movie/title "Mad Max Beyond Thunderdome"]
               [movie-eid :movie/director director-eid]]})

  (q '{:find [director-name]
       :where [[movie-eid :movie/title "Mad Max Beyond Thunderdome"]
               [movie-eid :movie/director director-eid]
               [director-eid :person/name director-name]]})

;
  )
and here's the XTDB 2 EA code:
(ns rlws.xtdb2
  (:require [xtdb.client :as xt.client]
            [xtdb.api :as xt]))

(comment

  (def xtdb-node (xt.client/start-client ""))
  (xt/status xtdb-node)

  (def people
    [{:name "George Miller", :id -142}
     {:name "George Ogilvie", :id -147}])

  (defn person->put-person [{:keys [id name] :as person}]
    [:put :person {:xt/id id, :name name}])

  (doseq [person people]
    (xt/submit-tx xtdb-node [(person->put-person person)]))

  (def movies
    [{:title "Mad Max Beyond Thunderdome", :director [-142 -147], :id -218}])

  (defn movie->put-movie [{:keys [id title director] :as movie}]
    [:put :movie {:xt/id id, :title title, :director director}])

  (doseq [movie movies]
    (xt/submit-tx xtdb-node [(movie->put-movie movie)]))

  (xt/q xtdb-node '{:find [title]
                    :where [($ :movie [title])]})

  (xt/q xtdb-node '{:find [director-eid]
                    :where [($ :movie {:title "Mad Max Beyond Thunderdome", :director director-eid})]})

  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  ;; THIS DOES NOT WORK
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  (xt/q xtdb-node '{:find [director-name]
                    :where [($ :movie {:title "Mad Max Beyond Thunderdome", :director director-eid})
                            ($ :person {:xt/id director-eid, :name director-name})]})

  ;
  )
The last query in the XTDB 2 EA code does not work! How do I do something like the last query in the XTDB 1.x code? Do we now need to create MANY-TO-MANY join tables? Creating a MANY-TO-MANY join table works just fine. Here's some XTDB 2 EA code that demonstrates it:
(ns rlws.xtdb2
  (:require [xtdb.client :as xt.client]
            [xtdb.api :as xt]))

(comment

  (def xtdb-node (xt.client/start-client ""))
  (xt/status xtdb-node)

  (def people
    [{:name "George Miller", :id -142}
     {:name "George Ogilvie", :id -147}])

  (defn person->put-person [{:keys [id name] :as person}]
    [:put :person {:xt/id id, :name name}])

  (doseq [person people]
    (xt/submit-tx xtdb-node [(person->put-person person)]))

  (def movies
    [{:title "Mad Max Beyond Thunderdome", :id -218}])

  (defn movie->put-movie [{:keys [id title] :as movie}]
    [:put :movie {:xt/id id, :title title}])

  (doseq [movie movies]
    (xt/submit-tx xtdb-node [(movie->put-movie movie)]))

  (def movie-directors
    [{:movie-id -218, :person-id -142, :id -300}
     {:movie-id -218, :person-id -147, :id -301}])

  (defn movie-director->put-movie-director [{:keys [id movie-id person-id] :as movie-director}]
    [:put :movie-director {:xt/id id, :movie-id movie-id, :person-id person-id}])

  (doseq [movie-director movie-directors]
    (xt/submit-tx xtdb-node [(movie-director->put-movie-director movie-director)]))

  (xt/q xtdb-node '{:find [person-name]
                    :where [($ :movie {:xt/id movie-id, :title "Mad Max Beyond Thunderdome"})
                            ($ :movie-director {:movie-id movie-id, :person-id person-id})
                            ($ :person {:xt/id person-id, :name person-name})]})

;
  )

👍 2
refset22:08:36

Hey Paul, sorry I've not been able to dig into this and look at your code properly yet or attempt to run it(!) My initial take is that join tables aren't inherently a bad idea, so you should be okay to use them, but I don't understand why your non-working example wouldn't work. Does it simply give the wrong result? I'll trying giving it a spin in the morning

hairfire13:08:50

Hey @U899JBRPF the query that does not work:

(xt/q xtdb-node '{:find [director-name]
                    :where [($ :movie {:title "Mad Max Beyond Thunderdome", :director director-eid})
                            ($ :person {:xt/id director-eid, :name director-name})]})
returns
[]

hairfire14:08:16

Any new thoughts on this?

refset15:08:57

so I was just about to post about running an 'office hours' session today starting in ~20m - this would be a good candidate 🙂 no need to attend if you're busy though, I'll post the solution

refset15:08:27

thanks again for your patience though, I will get back to you on this in hour or two, one way or the other(!)

malcolmsparks18:08:24

@me1408 that's a good question. One suggestion I have is that you write your transaction function as a SCI script you can store in the database. That way, you only ever need a single transaction function that knows how to call sci.core/eval-string. I've spoken about this in a talk here: https://youtu.be/Cq4S7nO1KOo