Clojurians Log v2
xtdb
It looks like https://github.com/xtdb/xtdb/issues/3894 has reappeared in a recent 2.0 SNAPSHOT.
Thanks for the report. Was there at least another log message suggesting that a crash log had been recorded? (Per https://github.com/xtdb/xtdb/issues/4155) are you you able to reproduce it?
Same repro as before: just run the next.jdbc test suite multiple times. It can't even get through the regular CI process (which runs the test suite three times, against Clojure 1.10, 1.11, and 1.12), so I've had to disable XTDB support for now just to get CI builds to complete.
I pasted everything from the XTDB console log to that ticket #3894, as well as the stacktrace etc.
Ah, sorry for causing trouble! We will investigate soon and report back
Hello 👋 I was looking for xtdb's take on data modeling and couldn't find anything (there was a promising discussion title: "https://github.com/orgs/xtdb/discussions/1631" but the link is broken). What is xtdb take on data modeling? is it the same as the relational SQL model?
Hey @lidorcg I'm confused what happened to that Discussions post 😅 in any case, this thread has some answers https://discuss.xtdb.com/t/data-modeling-question/327 tl:dr; in v2 you should feel able to treat it like Postgres, except you have better handling of nested data (without having to resort to JSONB when you do need to work with it)
v1 question: is there a way to configure the ingester? e.g. size of batches fetched from SQL db while building the RocksDB index?
Hey @thomasmoerman there are a few knobs available already. e.g. https://github.com/xtdb/xtdb/blob/75dc0901e2e3b805463a4523e544d9594c627b01/core/src/xtdb/tx.clj#L513 and https://github.com/xtdb/xtdb/blob/75dc0901e2e3b805463a4523e544d9594c627b01/modules/jdbc/src/xtdb/jdbc.clj#L232 and we could also consider exposing additional parameters to the module config paradigm, e.g. https://github.com/xtdb/xtdb/blob/75dc0901e2e3b805463a4523e544d9594c627b01/modules/jdbc/src/xtdb/jdbc.clj#L152 Are you trying to improve the performance of the rebuild to reduce some infra bottleneck? What is the objective?
In a way, when a fresh local dev env is connected to a "prod" Postgres (doc store), it taks quite a while (20-30 minutes) to rebuild the local RocksDB indexes on a fresh start. There are about half a million txs in the DB to consume. We use local rocksDB indexes, not a standalone xtdb instance, so every app update incurs a restart cost of a few minutes, which is fixable using checkpointing which I'm about to add. An app restart takes about 3 minutes on AWS/k8s to rebuild the indexes. At juxt, how do you guys typically run XT v1? As a standalone node or embedded?
We generally use v1 embedded, but yes it can definitely complicate deployments if you're running a complex/monolithic server
Checkpointing should help a lot, best to get that working first I reckon