Fork me on GitHub
#xtdb
<
2021-10-17
>
pinkfrog15:10:56

My scenario is like this: I have some pending tasks stored in xtdb, and have two consumers concurrently fetch tasks from xtdb and execute them. The requirement is that, one task can be only executed once. How can I ensure this ACID requirement with xtdb?

refset16:10:36

This might be over-kill for your needs but you could potentially adapt this https://github.com/ivarref/yoltq If you only have comparatively simple requirements though (e.g. no need for retry tracking), then I think you could use a transaction function to atomically check-then-lock a task to a single consumer

pinkfrog11:10:54

Hi refset, transaction function happens at each node during indexing. So the side effect may happen multiple times, which is not acceptable.

refset11:10:25

The side-effect wouldn't happen inside the transaction function - you would need the consumer to perform the side-effect after seeing that it has successfully locked the task. The consumer would then also update the task to show that it has been completed. You would probably want a max-time-bound to cope with consumer failure & retry though, and perhaps a mechanism to ensure that the consumer's local clock is ~accurate. You can also use match + backoff logic for this, but it will likely create too much contention and noise between the consumers.

refset11:10:16

Does yoltq look promising to you? Too much effort to adapt?

pinkfrog13:10:27

For the check-then-lock pattern, isn’t using match to verify the item is not locked the most idiomatic way?

pinkfrog13:10:38

btw, what’s the xtdb equivalent of the tx queue?

refset13:10:09

match is fine, it just might be lower throughput, although I'm not certain so it would be good to benchmark anyway - I might be completely wrong and actually match is way faster for only two consumers 🙂

refset13:10:23

> what’s the xtdb equivalent of the tx queue? have you seen open-tx-log?

pinkfrog14:10:57

I am not sure about how https://github.com/ivarref/yoltq is implemented. But I guess it lacks the consumer group support as in the following pattern:

xtdb -> single fetcher -> redis stream -> multiple consumers in the same consumer group

refset17:10:04

It looks vaguely like the diagram in the link from the yoltq readme