xtdb 2023-05-29 | Slack Archive

CarnunMP14:05:09

Quick gut check: I'm migrating from one db to another. Both are jdbc-all-the-things: tx log, document store, etc. Any reason why a pg_dump/pg_restore of the underlying Postgres db wouldn't do the trick? For one. For two, separately: any reason I shouldn't do something like query the history of every entity/document in the first db, filtering out duplicates (by :xtdb.api/content-hash), and transact those (with the original xtdb.api/tx-times) into the second db instead? I noticed I have a lot of duplicate documents in the first db is all, because I've been overzealously transacting thus far. But I reckon this migration is an opportunity to drop some dead weight. That said, the db is still pretty small (`tx_events` has ~60k rows in Postgres) and this is very likely overkill. But fun overkill, so long as it's not really really silly. 😜 Thoughts?

CarnunMP14:05:47

I suppose that if I went with the first option, after the fact I could swing back and evict the duplicates anyhow... if I really wanted to. Hmm.

CarnunMP14:05:17

#Also sent to the channel

Does https://docs.xtdb.com/language-reference/datalog-transactions/#evict delete rows in the underlying stores?

CarnunMP14:05:26

#Also sent to the channel

Huh. https://github.com/xtdb/xtdb/blob/8079bec1ae3e3d37e436b91e485205f2f2229132/modules/jdbc/src/xtdb/jdbc.clj#L112-L113

tatut14:05:31

idk, I wouldn’t touch tx_events and expect your db to remain consistent

CarnunMP14:05:17

Not planning on touching tx_events in any case. :))

tatut14:05:57

ah, I misunderstood that you wanted to partially transfer tx_events to another db

👍 2

CarnunMP14:05:41

I'm definitely bikeshedding or yak shaving or whatever the term is, here. But fun to think about.

🙂 2

refset22:05:51

> https://github.com/xtdb/xtdb/blob/8079bec1ae3e3d37e436b91e485205f2f2229132/modules/jdbc/src/xtdb/jdbc.clj#L112-L113 tombstones are likely much smaller than whatever maps you put in originally, but yeah no rows get deleted https://github.com/xtdb/xtdb/blob/8079bec1ae3e3d37e436b91e485205f2f2229132/core/src/xtdb/kv/index_store.clj#L1113-L1114

👍 2

CarnunMP14:05:17

replied to a thread:Quick gut check: I'm migrating from one db to another. Both are jdbc-all-the-things: tx log, document store, etc. Any reason why a pg_dump/pg_restore of the underlying Postgres db wouldn't do the trick? For one. For two, separately: any reason I shouldn't do something like _query the history of every entity/document_ in the first db, filtering out duplicates (by `:xtdb.api/content-hash`), and transact those (with the original `xtdb.api/tx-time`s) into the second db instead? I noticed I have a lot of duplicate documents in the first db is all, because I've been overzealously transacting thus far. But I reckon this migration is an opportunity to drop some dead weight. That said, the db is still pretty small (`tx_events` has ~60k rows in Postgres) and this is very likely overkill. But _fun_ overkill, so long as it's not really really silly. :stuck_out_tongue_winking_eye: Thoughts?

Does https://docs.xtdb.com/language-reference/datalog-transactions/#evict delete rows in the underlying stores?

CarnunMP14:05:26

replied to a thread:Quick gut check: I'm migrating from one db to another. Both are jdbc-all-the-things: tx log, document store, etc. Any reason why a pg_dump/pg_restore of the underlying Postgres db wouldn't do the trick? For one. For two, separately: any reason I shouldn't do something like _query the history of every entity/document_ in the first db, filtering out duplicates (by `:xtdb.api/content-hash`), and transact those (with the original `xtdb.api/tx-time`s) into the second db instead? I noticed I have a lot of duplicate documents in the first db is all, because I've been overzealously transacting thus far. But I reckon this migration is an opportunity to drop some dead weight. That said, the db is still pretty small (`tx_events` has ~60k rows in Postgres) and this is very likely overkill. But _fun_ overkill, so long as it's not really really silly. :stuck_out_tongue_winking_eye: Thoughts?

Huh. https://github.com/xtdb/xtdb/blob/8079bec1ae3e3d37e436b91e485205f2f2229132/modules/jdbc/src/xtdb/jdbc.clj#L112-L113

CarnunMP17:05:46

Hmm. For some reason I haven't been able to work out, I'm getting an IllegalArgumentExeption when I try to provide an ::xt/tx-time to submit-tx (on a fresh node):

(xt/submit-tx node [[::xt/put doc]] {::xt/tx-time tx-time})
; => Syntax error (IllegalArgumentException) compiling xt/submit-tx at (carnun/com/db/db.clj:120:5).                                                                  
;    No single method: submit_tx of interface: xtdb.api.PXtdbSubmitClient found for function: submit-tx of protocol: PXtdbSubmitClient
;    Clojure: class clojure.lang.Compiler$CompilerException

If I omit the map with ::xt/tx-time, the transaction goes through just fine. Any ideas?

CarnunMP17:05:47

Quite sure I'm good re the second condition here: ⬇️ But maybe the first is violated when there haven't yet been any transactions? :thinking_face:

refset22:05:10

hey @U018D6NKRA4 was this with new-submit-client or a regular start-node node*?*

CarnunMP10:06:50

the latter

refset11:06:23

that's bizarre, I can't repro

(let [n (xt/start-node {})]
  (xt/submit-tx n [[::xt/put {:xt/id :foo}]] {::xt/tx-time #inst "1979-01-01"}))

refset11:06:54

are you able to hit this consistently? which XT/Clojure/Java version are you using?

CarnunMP15:06:20

$ java --version
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10)
OpenJDK 64-Bit Server VM (build 17.0.6+10, mixed mode)

{...

 :deps
 {org.clojure/clojure {:mvn/version "1.11.0-rc1"}

  ...

  ;; db
  com.xtdb/xtdb-core {:mvn/version "1.20.0"}
  com.xtdb/xtdb-jdbc {:mvn/version "1.20.0"}
  com.github.seancorfield/next.jdbc {:mvn/version "1.2.772"}
  org.postgresql/postgresql {:mvn/version "42.3.3"}

  ...
  }}

CarnunMP15:06:21

as for the former, will have to swing back in a bit to answer! because https://clojurians.slack.com/archives/CG3AM2F7V/p1685394042458079 and I'm now chasing down a separate issue (a memory leak apparently in/involving XT — will post here if so & necessary 🙂 )

CarnunMP16:06:10

(huh, not sure how, but bumping xt to 1.23.2 seems to have fixed the memory leak!)

CarnunMP16:06:59

I can confirm @U899JBRPF that (xt/submit-tx n [[::xt/put {:xt/id :foo}]] {::xt/tx-time #inst "1979-01-01"}) produces the IllegalArgumentException on 1.20.0 but not latest, 1.23.2!

CarnunMP16:06:23

I should have checked that first, my bad :))

refset14:06:17

(just returning after a long vacation, apologies for the delay) ah, thanks for confirming that - I can confirm that tx-time import/override has only been available since https://github.com/xtdb/xtdb/releases/tag/1.21.0 😅

👍 2

😁 2

CarnunMP21:05:42

Fwiw, I went the pg_dump/`pg_restore` route in the end. :))

2023-05-29

Channels