This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
My scenario is like this: I have some pending tasks stored in xtdb, and have two consumers concurrently fetch tasks from xtdb and execute them. The requirement is that, one task can be only executed once. How can I ensure this ACID requirement with xtdb?
This might be over-kill for your needs but you could potentially adapt this https://github.com/ivarref/yoltq If you only have comparatively simple requirements though (e.g. no need for retry tracking), then I think you could use a transaction function to atomically check-then-lock a task to a single consumer
Hi refset, transaction function happens at each node during indexing. So the side effect may happen multiple times, which is not acceptable.
The side-effect wouldn't happen inside the transaction function - you would need the consumer to perform the side-effect after seeing that it has successfully locked the task. The consumer would then also update the task to show that it has been completed. You would probably want a max-time-bound to cope with consumer failure & retry though, and perhaps a mechanism to ensure that the consumer's local clock is ~accurate.
You can also use match
+ backoff logic for this, but it will likely create too much contention and noise between the consumers.
For the check-then-lock pattern, isn’t using match
to verify the item is not locked the most idiomatic way?
match
is fine, it just might be lower throughput, although I'm not certain so it would be good to benchmark anyway - I might be completely wrong and actually match
is way faster for only two consumers 🙂
I am not sure about how https://github.com/ivarref/yoltq is implemented. But I guess it lacks the consumer group support as in the following pattern:
xtdb -> single fetcher -> redis stream -> multiple consumers in the same consumer group