Fork me on GitHub
#xtdb
<
2023-04-16
>
tatut06:04:21

had a some weird failure mode that queries from local index found things but the documents came up completely empty when pulling them, restarting the nodes fixed it (using postgresql for tx-log and document store, rocksdb for local indexes)

tatut06:04:47

I’ll try to peruse the logs later, if I can find some jdbc connection related problems… but I would expect pull to fail if it can’t get the doc

tatut07:04:40

not seeing any exception from xtdb

refset07:04:52

Ah, this sounds like https://github.com/xtdb/xtdb/issues/1897 - we have broadly addressed these document store failure modes on master already now (so I think the issue can be closed, but I'll double check tomorrow) and the next release will happen in a few days. In the meantime you could try building master if you are particularly concerned about the potential impact for your app

tatut07:04:29

ok, good to know… this was a little scary, but I think I could add some pull&verify check to health endpoint in the mean time

tatut07:04:51

we are using a “serverless” rds postgres on aws, which will scale down when not in use, so this morning nothing was being pulled… but there was no log entry anywhere about any sql query timeout or failure

tatut07:04:26

Can I do some check in user space, that validates doc store connection and doesn’t use the doc cache

wotbrew13:04:58

@U11SJ6Q0K can I ask what XTDB version you were running when you observed this?

tatut13:04:57

1.23.1

👍 2