For my use case, I'm trying to store about 15k email threads in xtdb that I can query. Each email thread has one or more messages that I intended to store in an ordered list. Based off the docs, it seems like you can have an attribute that is a set of references, but I couldn't find any guidance for having an attribute that is an ordered list of references.

Hm, you could of course separate each email as docs with a reference to the parent

Or just store them in a vector?

But if you want to query anything inside the messages, they need to be distinct documents/entities

The referencing works out of the box. It's the ordered part that I was looking for more info.

Vector of references in the parent?

Though that means the parent changes every time a child is added, which is suboptimal

Children could also store their index, which could be enforced if the emails are added with a transaction function

In my initial tests, it seemed like if you used a vector as the value for an attribute, it would treat it as an unordered set

You can still retrieve the original vector in the query, either via the pull syntax, or by calling xt/entity from within the query

The ordering is actually stored deep in the index also, but it's invisible to the basic Datalog semantics

:thumbsup: ah, good to know. I had trouble telling if that was the case from the docs

Searching for anything related to ordering is complicated by the fact that many examples use storing "orders" as their example domain.

Are there any other docs on multi-valued attributes? The only mention I could find was https://docs.xtdb.com/language-reference/datalog-queries/#_query_join_with_two_attributes_including_a_multi_valued_attribute

Not really... basically it only works with Clojure sets and Clojure vectors

There is a test showing use of sets here: https://github.com/xtdb/xtdb/blob/master/test/test/xtdb/query_test.clj#L1188-L1212

Given everything works with set semantics, any repeated values/IDs in a vector will be treated accordingly

Assuming that there are no duplicates, order is guaranteed to be preserved?

Now I'm confused

(xt/submit-tx xtdb-node [[::xt/put
                          {:xt/id "parent"
                           :my/children ["child1" "child2" "child1"]}
                          ]
                         [::xt/put
                          {:xt/id "child1"
                           :my/parent "parent"}]
                         [::xt/put
                          {:xt/id "child2"
                           :my/parent "parent"}]])


> (xt/q (xt/db xnode) '{:find [(pull ?e [{:my/children [:xt/id :my/parent]}])]
                      :where [[?e :xt/id "parent"]]})
;; #{[#:my{:children 
  [{:my/parent "parent", :xt/id "child1"} 
   {:my/parent "parent", :xt/id "child2"} 
   {:my/parent "parent", :xt/id "child1"}]}]}

This behavior is basically what I was looking for, but I'm not sure if I'm relying on undefined behavior.

Great - that is definitely intended behaviour - pull bypasses the indexes (which would mean reconstructing the triples) and instead returns the original document (via a cache, and whatever filtering / pull transformations you apply)

> order is guaranteed to be preserved? Once documents are accepted and serialized via ::xt/put (using Nippy under the hood to serialize in docs in their entirety) they are guaranteed to be retrievable as-is - barring any fundamental bugs in Nippy or the tx-log/doc-stores

I haven't used mount before, so bear with me, but I guess something isn't being deref'd as it should 🙂 what's the definition of start-xtdb!?

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir      (io/file dir)
                        :block-cache :xtdb.rocksdb/block-cache
                        :sync?       true}})]
    (doto                                                   ;
      (xt/start-node 
                     {:xtdb/tx-log              (kv-store "data/dev2/tx-log")
                      :xtdb/document-store      (kv-store "data/dev2/doc-store")
                      :xtdb/index-store         (kv-store "data/dev2/index-store")
                      ;; Configuring the Block Cache 
                      :xtdb.rocksdb/block-cache {:xtdb/module 'xtdb.rocksdb/->lru-block-cache
                                                 :cache-size  (* 512 1024 1024)}
                      ;; optional:
                      :xtdb.lucene/lucene-store {:db-dir "data/dev2/lucene-dir"}
                      :xtdb.http-server/server  {:port 9999}
                      }))
    xt/sync)
  )

Im running out from ideas unfortunately.. so if you (anybody) have any idea.. Im really happy.

Since this post, I tried to add `

(defstate ^:dynamic ^IXtdb *xtdb*

form repl (run mount/start) work everything fine.. so therefore I don’t understand this now..

Not sure if it's relevant, but I'm running M1 mac

Update: i didnt read in the night accurately the post. Therefore I delete this

UPDATE: ah, I just read again your original message and it looks like you already tried purging and trying different directory so my original comment below doesn't help much. It looks like you have another xtdb node instance running in the same dir. You need to run (.close node) on the node instance to close it - although it's easy to loose a handle to it in the REPL. In your data/dev/ dir there's probably an unreleased LOCK file - to test it you can either purge the dir (if you're sure you have no valuable data there) or specify another dir location in the xtdb.api/start-node arg.

@UHA0AQZ2M, thanks for taking a look. Your comment made me double check myself and I found the issue. It turns out I also used the exact same snippet in another local project that is a dependency. When I loaded that dependency's namespace, it was recreating the db in the same dir before my current namespace tried to do the same.