xtdb 2022-10-11 | Slack Archive

macrobartfast03:10:09

Is there an easy way to evict all documents from the db? I am used to wildcards in other dbs but am not sure how that works here or if it’s applicable. I know how to evict one document, so I suppose I could write a function that queries and returns all the xt/id’s and then delete them one by one… but maybe there’s a faster way. UPDATE: I was able to map an eviction function against the results of a full query and it did the trick.

Jacob O'Bryant17:10:43

If you want to evict all the documents, maybe just do it on the storage side? i.e. if you're using the filesystem for tx log + doc store, just delete those directories (and delete the index). if you're using postgres, delete the database. etc otoh if the number of documents is small enough that eviction doesn't take long, then eh ¯\(ツ)/¯

➕ 1

macrobartfast18:10:31

That would work. Mapping an eviction function across all the id’s and it seemed to work fine. I’m glad it seems to take a few steps…

👌 1

macrobartfast18:10:16

madstap22:10:38

Can I safely do a side effect inside a transaction function to guarantee that if it's in the db it happened at-least-once?

{:xt/fn
 '(fn [node kafka-producer]
    (produce! kafka-producer ,,,)
    [[::xt/put {:xt/id 1 :written? true}]])}

tatut04:10:34

don’t the tx fn arguments need to be serializable, which a kafka producer connection wouldn’t be?

✔️ 1

tatut04:10:34

I wouldn’t have any outside IO in tx functions… perhaps you could use a tx listener that listens to the right kind of transactions and produces those to kafka?

➕ 2

jarohen08:10:11

also bear in mind that every node plays through the transaction log, so if you have a multi-node system it'll happen multiple times. (edit: although you did say at-least-once)

jarohen09:10:17

and also that even if you have a single-node system it may well need to replay the log (in the case of an index version upgrade) which will then run all of your side effects again

jarohen09:10:00

even if multiple invocations wasn't a problem for you, the IO could fail on some nodes but not others - in which case the transaction would no longer consistently commit or rollback across the cluster, and you'd end up with a different state on different nodes (or after replay)

madstap18:10:21

Got it, not a good idea, thanks

refset10:10:40

I think the only justifiable exceptions here are for observability/logging

2022-10-11

Channels