This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-01-25
Channels
- # announcements (4)
- # babashka (58)
- # beginners (21)
- # calva (42)
- # clj-kondo (15)
- # cljdoc (16)
- # cljs-dev (11)
- # clojure (57)
- # clojure-denmark (1)
- # clojure-europe (24)
- # clojure-france (4)
- # clojure-nl (1)
- # clojure-norway (16)
- # clojure-spec (6)
- # clojure-uk (2)
- # clojurescript (19)
- # clr (4)
- # conjure (1)
- # core-logic (3)
- # cursive (5)
- # data-science (2)
- # datascript (6)
- # datomic (3)
- # emacs (4)
- # events (3)
- # fulcro (17)
- # gratitude (2)
- # hyperfiddle (4)
- # introduce-yourself (3)
- # jobs-discuss (2)
- # lsp (27)
- # malli (22)
- # pathom (73)
- # portal (21)
- # re-frame (5)
- # releases (1)
- # vim (8)
- # xtdb (28)
Hey, run into funny sync
related issue. Local indices rocksdb, db is a remote PostgreSQL.
My xt
node
{:xtdb.version/version "1.20.0",
:xtdb.version/revision "421db6dfe3bbcc57dac2db8eeafb9ece2a4703d5",
:xtdb.index/index-version 18,
:xtdb.tx-log/consumer-state nil,
:xtdb.kv/kv-store "xtdb.rocksdb.RocksKv",
:xtdb.kv/estimate-num-keys 4598991,
:xtdb.kv/size 168408771}
Where as the remote db is already been used with 1.23.0
version. So, I tried to sync
my local indices and got the following error after a minute or two (our db is large)
Execution error (IllegalArgumentException) at xtdb.tx.conform/conformed-tx-events->doc-hashes (conform.clj:216).
No method in multimethod '<-tx-event' for dispatch value: :xtdb.tx.event/tx-events
Any ideas what could be the reason? Colleague with a newer xt
node version (1.23.0) also has much larger estimate with the same remote db instance.
I'm going to update my node version soonish and expect this to disappear, but this raises some concerns for future version bumps/migrations.After updating xtdb
version, I got a more meaningful error message with the old index.
; Execution error (IndexVersionOutOfSyncException) at xtdb.kv.index-store/check-and-store-index-version (index_store.clj:357).
; Index version on disk: 18 does not match index version of code: 22
Thinking now that if we are updating multiple nodes with old indices and we upgrade xtdb
version, we might run into a situation where different versions try to access db with different index versions.
hey @U0267SNCPEY π nodes < 1.21.0 https://github.com/xtdb/xtdb/issues/1888 to read transaction logs with transactions submitted after 1.21.0 due to the new ability for the user to explicitly specify the tx-time of a transaction (for bulk import purposes) - we're investigating what's involved in a fix, but in the meantime I'd recommend upgrading the nodes (as you have)
So I could assume that after this update we shouldn't run into similar incompatibilities or they would labeled as breaking changes?
> After updating xtdb
version, I got a more meaningful error message with the old index
This is an 'index version bump' - we put these through every now and again when we want to upgrade the index structure - most recently a handful of indexing performance improvements. In this case, you'll want to let a new version of XT re-index from the tx-log in the background, and create a checkpoint that other nodes can then use

> Thinking now that if we are updating multiple nodes with old indices and we upgrade xtdb
version, we might run into a situation where different versions try to access db with different index versions.
each node has its own index, so you shouldn't ever get a case where a node reads an incompatible index version - that's what the 'index version' check prevents
> So I could assume that after this update we shouldn't run into similar incompatibilities or they would labeled as breaking changes? yep, absolutely. tbh this is the only tx-log format change we've put in in my ~3 years in XTDB - it's not something we do on a regular basis, for this very reason π
> each node has its own index, so you shouldn't ever get a case where a node reads an incompatible index version - that's what the 'index version' check prevents So, it seems that my case simply reduces to a case where I couldn't sync my local indices as my local index had too old version and the remote db had been updated with newer XTDB version π
fwiw we're looking into the possibility of a non-breaking patch release in the 1.20.x line that users could upgrade to that then would be able to handle >=1.21.0 transactions if that would be useful

It would allow first upgrading all nodes to 1.20.X to be able to read the logs, and then to 1.21.0 so that at no point there would be nodes writing transactions while nodes that cannot read them are still online
More the reason to have a staging environment that's identical to the real one, including having multiple nodes
Though I suppose it's a bit tricky to catch even then. You'd need the new nodes to insert new transactions in the middle of the upgrade. Or even exactly the right kind of transactions?
Is it possible to both update an entity and delete an entity in the same transaction? so that the updated entity is available in the entity-history?
First attempt suggests no
you can if they have different valid time ranges - but if they have both the same valid-time range and tx-time they'll cancel each other out
but yeah, more generally you can't have a zero-width valid-time range in XTDB - it's inclusive-start exclusive-end
[#:xtdb.api{:content-hash #xtdb/id "0000000000000000000000000000000000000000"
:doc nil
:tx-id 2
:tx-time #inst "2023-01-25T18:06:23.447-00:00"
:valid-time #inst "2023-01-25T18:06:23.448-00:00"}
#:xtdb.api{:content-hash #xtdb/id "c628864629963b3311fadad544d1d1071a5edc05"
:doc {:widget/created-at #inst "2023-01-25T18:06:23.418-00:00"
:widget/created-by #:nexus.user{:id #uuid "00000000-0000-0000-0000-000000000001"}
:widget/deleted-at #inst "2023-01-25T18:06:23.447-00:00"
:widget/deleted-by #:nexus.user{:id #uuid "00000000-0000-0000-0000-000000000001"}
:widget/id #uuid "0185ea1c-06ba-bd41-af80-b84d143e1257"
:widget/label "Label 3"
:xt/id #:widget{:id #uuid "0185ea1c-06ba-bd41-af80-b84d143e1257"}}
:tx-id 2
:tx-time #inst "2023-01-25T18:06:23.447-00:00"
:valid-time #inst "2023-01-25T18:06:23.447-00:00"}
#:xtdb.api{:content-hash #xtdb/id "a25ca7c8f2a07af6aaf18019d7312833f9feb453"
:doc {:widget/created-at #inst "2023-01-25T18:06:23.418-00:00"
:widget/created-by #:nexus.user{:id #uuid "00000000-0000-0000-0000-000000000001"}
:widget/id #uuid "0185ea1c-06ba-bd41-af80-b84d143e1257"
:widget/label "Label 3"
:xt/id #:widget{:id #uuid "0185ea1c-06ba-bd41-af80-b84d143e1257"}}
:tx-id 1
:tx-time #inst "2023-01-25T18:06:23.418-00:00"
:valid-time #inst "2023-01-25T18:06:23.418-00:00"}]
sorry if this question is naive, but, does xtdb fit the definition of 'time series graph database' ? My understanding of 'TS Graph D' is one where I can do typical graph analysis and typical timeseries functions [aggregations, data summarisations over time] out of one engine/schema. I just wanted to get the terminology right. thank you
hey @U04HFRYJTNW π we don't tend to describe ourselves as a time-series database - this tends to bring a few expectations that we don't aim to cover, e.g. windowing, retention etc. graph: more so - we're a document database which you can query using a graph-like query language, interpreting the values in the documents as potential edges.
it's not a true 'property graph' model - i.e. modelled as distinct vertices and edges - if, for example, you want to add properties to your edges, we recommend that you make another document for the edge with additional attributes, referencing the vertex documents - but we think it covers enough of the benefits with a simpler model π
I saw this comparison a few months back that might add some clarity on the 'time series' aspect https://www.pgcon.org/2019/schedule/track/Applications/1336.en.html