Fork me on GitHub
#xtdb
<
2023-08-25
>
refset14:08:12

I recorded a short video with @danstone yesterday discussing some REPL workflow thoughts - not so much focused on XT itself, but it might be of interest! https://twitter.com/xtdb_com/status/1695072870089589060

clojure-spin 3
❤️ 1
richiardiandrea16:08:00

I have a quick question, in dev mode sometimes we "wipe" the db (`tx_events`) included. We have an XTDB listener running and when we do that the connection poller (`xtdb.tx.subscribe$handle_polling_subscription`) is throwing and does not seem to retry "enough". Is there a knob for it to configure there? The actual error I get, after recreating tx_events is > xtdb.tx.subscribe - Error polling for txs, will retry > java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

richiardiandrea17:08:19

This is actually a problem in our scripts it seems, we down the docker container containing postgres

refset17:08:22

Hey @U0C8489U6 ! So you're expecting the node to be able to resume seamlessly somehow? Doing it naively wouldn't be safe if you are actually modifying tx_events out-of-band - like if you wipe the table then start submitting new transactions you're going to get into a pretty funky state

refset17:08:56

ah, so you really want to wipe the local node state (index-store) also - is cycling the node(s) not possible?

richiardiandrea17:08:23

yeah cause in dev that's a common flow for us

👌 1
richiardiandrea17:08:44

dropping containers for now it is not working - that's on us, tech debt

👍 1
refset17:08:53

well, I have considered pushing the team for a dev-time-only drop! API before but we've never spec'd it out or decided that it's definitely a bad idea - in lieu of that I'm struggling to think of other options

richiardiandrea17:08:38

another good use case for that for us is for E2E testing, we reset postgres before re-seeding with the data we need (hitting the apis)

👍 1
refset17:08:06

back to the original question more directly, I suspect it would be non-trivial for a node to somehow know for sure that it should 'reset' itself based on a wiped/modified tx_events - at least without introducing a new coordination mechanism (e.g. store some UUID at the beginning of the log that tracks such epoch changes)

refset17:08:47

assuming you're embedding XT nodes in your own code anyway I guess you can at least hack something together for this in userspace

👍 1
richiardiandrea17:08:59

I am thinking maybe restarting the XTDB component might work? given we wipe that table plus all the checkpointing dirs?

refset17:08:08

if there are no checkpoints and you haven't configured a durable kv store (Rocks / LMDB) then yep you should be okay to restart your component blob_thumbs_up

richiardiandrea17:08:01

right, didn't think about the kv store...

refset18:08:05

well I have definitely successfully used clojure.java.shell/sh to rm -rf the directories before :)

👍 1