Fork me on GitHub
#xtdb
<
2022-05-25
>
phronmophobic17:05:46

Are there any docs or examples that cover how to work with storing and updating ordered lists?

phronmophobic18:05:56

For my use case, I'm trying to store about 15k email threads in xtdb that I can query. Each email thread has one or more messages that I intended to store in an ordered list. Based off the docs, it seems like you can have an attribute that is a set of references, but I couldn't find any guidance for having an attribute that is an ordered list of references.

Hukka18:05:53

Hm, you could of course separate each email as docs with a reference to the parent

Hukka18:05:00

Or just store them in a vector?

Hukka18:05:27

But if you want to query anything inside the messages, they need to be distinct documents/entities

phronmophobic18:05:12

The referencing works out of the box. It's the ordered part that I was looking for more info.

Hukka18:05:31

Vector of references in the parent?

Hukka18:05:50

Though that means the parent changes every time a child is added, which is suboptimal

Hukka18:05:34

Children could also store their index, which could be enforced if the emails are added with a transaction function

phronmophobic18:05:48

In my initial tests, it seemed like if you used a vector as the value for an attribute, it would treat it as an unordered set

refset18:05:53

You can still retrieve the original vector in the query, either via the pull syntax, or by calling xt/entity from within the query

refset18:05:10

The ordering is actually stored deep in the index also, but it's invisible to the basic Datalog semantics

phronmophobic18:05:31

:thumbsup: ah, good to know. I had trouble telling if that was the case from the docs

👌 1
phronmophobic18:05:24

Searching for anything related to ordering is complicated by the fact that many examples use storing "orders" as their example domain.

😄 1
refset18:05:12

Not really... basically it only works with Clojure sets and Clojure vectors

👍 1
refset18:05:27

Given everything works with set semantics, any repeated values/IDs in a vector will be treated accordingly

phronmophobic18:05:14

Assuming that there are no duplicates, order is guaranteed to be preserved?

phronmophobic18:05:15

Now I'm confused

(xt/submit-tx xtdb-node [[::xt/put
                          {:xt/id "parent"
                           :my/children ["child1" "child2" "child1"]}
                          ]
                         [::xt/put
                          {:xt/id "child1"
                           :my/parent "parent"}]
                         [::xt/put
                          {:xt/id "child2"
                           :my/parent "parent"}]])


> (xt/q (xt/db xnode) '{:find [(pull ?e [{:my/children [:xt/id :my/parent]}])]
                      :where [[?e :xt/id "parent"]]})
;; #{[#:my{:children 
  [{:my/parent "parent", :xt/id "child1"} 
   {:my/parent "parent", :xt/id "child2"} 
   {:my/parent "parent", :xt/id "child1"}]}]}

phronmophobic18:05:33

This behavior is basically what I was looking for, but I'm not sure if I'm relying on undefined behavior.

refset19:05:59

Great - that is definitely intended behaviour - pull bypasses the indexes (which would mean reconstructing the triples) and instead returns the original document (via a cache, and whatever filtering / pull transformations you apply)

refset19:05:00

> order is guaranteed to be preserved? Once documents are accepted and serialized via ::xt/put (using Nippy under the hood to serialize in docs in their entirety) they are guaranteed to be retrievable as-is - barring any fundamental bugs in Nippy or the tx-log/doc-stores

sb19:05:45

Could you help how it is possible fix this? Any idea?

java.lang.IllegalArgumentException: No implementation of method: :db of protocol: #'xtdb.api/DBProvider found for class: mount.core.DerefableState
        clojure.lang.ExceptionInfo: clojure.lang.ExceptionInfo in Interceptor :geheimtur.interceptor/interactive-auth - java.lang.IllegalArgumentException in Interceptor :geheimtur.impl.form-based/default-login-handler - No implementation of method: :db of protocol: #'xtdb.api/DBProvider found for class: mount.core.DerefableState

refset20:05:45

I haven't used mount before, so bear with me, but I guess something isn't being deref'd as it should 🙂 what's the definition of start-xtdb!?

sb22:05:31

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir      (io/file dir)
                        :block-cache :xtdb.rocksdb/block-cache
                        :sync?       true}})]
    (doto                                                   ;
      (xt/start-node 
                     {:xtdb/tx-log              (kv-store "data/dev2/tx-log")
                      :xtdb/document-store      (kv-store "data/dev2/doc-store")
                      :xtdb/index-store         (kv-store "data/dev2/index-store")
                      ;; Configuring the Block Cache 
                      :xtdb.rocksdb/block-cache {:xtdb/module 'xtdb.rocksdb/->lru-block-cache
                                                 :cache-size  (* 512 1024 1024)}
                      ;; optional:
                      :xtdb.lucene/lucene-store {:db-dir "data/dev2/lucene-dir"}
                      :xtdb.http-server/server  {:port 9999}
                      }))
    xt/sync)
  )

sb22:05:37

Im running out from ideas unfortunately.. so if you (anybody) have any idea.. Im really happy.

sb22:05:12

Since this post, I tried to add `

(defstate ^:dynamic ^IXtdb *xtdb*
IXtdb too.. because I saw in the JUXT repo.. but I don’t know.. that isn’t helped. What I don’t understand why run the http server on 9999.. where I can query the db without problems.. when I can’t access to the things from code. Maybe very trivial problem.. just I’m tired. 😉

sb22:05:45

form repl (run mount/start) work everything fine.. so therefore I don’t understand this now..

sb19:05:54

I use

(mount/defstate ^:dynamic *xtdb*
                :start (start-xtdb!)
                :stop (when (not (keyword? *xtdb*)) (.close *xtdb*)))

sb19:05:37

I get this error message after I run lein uberjar and java -jar app.jar and ofc at authentication as you see (first xtdb call), the xtdb.http-server/server works fine as I see on the port.

phronmophobic22:05:35

I seemed to have put my local dev system in a weird state. When I call start-xtdb! (which is just using the code from the getting started section):

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir (io/file dir)
                        :sync? true}})]
    (xt/start-node
     {:xtdb/tx-log (kv-store "data/dev/tx-log")
      :xtdb/document-store (kv-store "data/dev/doc-store")
      :xtdb/index-store (kv-store "data/dev/index-store")})))


(defonce gdb
  (start-xtdb!))
I get the following error:
Execution error (RocksDBException) at org.rocksdb.RocksDB/open (RocksDB.java:-2).
lock hold by current process, acquire time 1653517215 acquiring thread 6235123712: <project dir>/data/dev/index-store/LOCK: No locks available
The database didn't have anything in it so I tried quitting my repl, deleting the data dir, and starting over, but I still get the same error. Creating a database in a different directory succeeds. Is there a way to make that directory available again to store the db data?

phronmophobic22:05:58

Not sure if it's relevant, but I'm running M1 mac

sb22:05:08

Update: i didnt read in the night accurately the post. Therefore I delete this

Piotr Roterski23:05:25

UPDATE: ah, I just read again your original message and it looks like you already tried purging and trying different directory so my original comment below doesn't help much. It looks like you have another xtdb node instance running in the same dir. You need to run (.close node) on the node instance to close it - although it's easy to loose a handle to it in the REPL. In your data/dev/ dir there's probably an unreleased LOCK file - to test it you can either purge the dir (if you're sure you have no valuable data there) or specify another dir location in the xtdb.api/start-node arg.

phronmophobic00:05:10

@UHA0AQZ2M, thanks for taking a look. Your comment made me double check myself and I found the issue. It turns out I also used the exact same snippet in another local project that is a dependency. When I loaded that dependency's namespace, it was recreating the db in the same dir before my current namespace tried to do the same. facepalm

😂 1
🙌 1