Fork me on GitHub
#xtdb
<
2022-10-27
>
wotbrew16:10:52

1.22.1 has just been released! 🚀 It comes with two new temporal predicates to do a bit more with valid time in datalog queries, and a few bug fixes. See https://github.com/xtdb/xtdb/releases/tag/1.22.1 for more details. Thanks to all who raised issues, tested or otherwise contributed to this release 🙏. Special thanks to @tatut and @finn.volkel, you both rock! 😎 .

🎉 10
❤️ 2
walterl22:10:47

What's the Right Way to export data from an XTDB node, and import it in another, in a way that preserves all metadata? Context: I want to switch storage engines.

👀 1
refset22:10:53

There isn't a comprehensive migration utility available, but previously we've handled specific batch export & import flows like this (via .edn files): https://github.com/xtdb/xtdb/commit/efc5f93f372ecbf18445fbcd60c43359f0f07f86 https://github.com/xtdb/xtdb/commit/7999b8e17366a4cd3d6bd4dd66631052bdeb02c0 What is the tx-log/doc-store combo you're hoping to move from/to?

👀 1
walterl23:10:08

Thanks, checking out those links... I'm looking to swap out RocksDB for LMDB, mostly just to try it out and see how resource usage compares, and if the latter would support concurrent access from different processes.

👍 1
wotbrew09:10:09

Hey, not sure if this is helpful but, just so you aware LMDB kv stores in xtdb support many concurrent readers, but just one writer - see note on MDB_NOLOCK here: http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5d579e71f22e9340 There may be ways to work around this in the future but as it stands each kv-store instance opens an LMDB environment which has this restriction. You might be ok if you are talking about multiple OS processes each opening their own LMDB environment.

walterl12:10:24

Thanks for that detail, @U0GE2S1NH. I wasn't aware. 🙌 Yes, I was referring to multiple OS processes, so I'll test that 🤞

walterl12:10:34

This turned out to be sufficient for export/import in my use case. 🙂

(defn dump-txns
  "Dump XTDB transactions to EDN file."
  [node fname]
  (let [{:keys [lazy-seq-iterator close-fn]} (xt/open-tx-log node 0 true)
        pformat (fn [x] (with-out-str (pp/pprint x)))]
    (try
      (with-open [w (io/writer fname)]
        (->> (iterator-seq lazy-seq-iterator)
             (sort-by ::xt/tx-id)
             (pformat)
             (.write w)))
      (finally
        (close-fn)))))

(defn load-txns
  "Load XTDB transactions from EDN file"
  [node fname]
  (doseq [tx (with-open [r (io/reader fname)]
               (edn/read
                 {:readers {'xtdb/id xc/id-edn-reader}}
                 (java.io.PushbackReader. r)))
          :let [{::xt/keys [tx-time tx-ops] :as _tx} tx]]
    (xt/submit-tx node tx-ops {::xt/tx-time tx-time}))
  (xt/sync node))

refset12:10:44

Great, glad to hear and thanks for sharing that snippet @UJY23QLS1! I feel it's necessary to mention however that we do not currently make a guarantee that the output of open-tx-log can always correctly be piped into submit-tx like this (although in theory this would be ideal). If it works for your case right now then no problem but just making sure that it's clear 🙂

walterl12:10:41

Noted, thanks. 🙂

🙏 1