Back.

We have designed the output plugin interfaces to allow you to implement async requests like this in a sensible way.

You can see it in play in the sqs plugin.

https://github.com/onyx-platform/onyx-amazon-sqs/blob/0.12.x/src/onyx/plugin/sqs_output.clj#L59

prepare-batch allows you to get all of your requests together

then write-batch allows you to initiate your requests, and return false if you’re not done, or true if you are done

Generally I’d recommend returning true if you’ve started all of your requests, or false if you have only started some of them and you want to stay at the write-batch stage

Then the additional missing piece is that you HAVE to ensure all of your requests have completed in the synced? call https://github.com/onyx-platform/onyx-amazon-sqs/blob/0.12.x/src/onyx/plugin/sqs_output.clj#L44

The sync call is what allows a barrier to progress through, and if you have synced you are essentially signalling that you have completed everything between two barriers and it is safe to checkpoint.

The idea is that you have two places you can backpressure (write-batch and synced?). write-batch and prepare-batch also allow you to yield and not progress to the next state, which allows you to give back the thread to the peer so that it can heartbeat if your request is long running. That way your peer doesn’t get timed out.

Ah OK

So if write-batch returns false, the task will yield execution back to the thread? So basically I can do something like the http plugin that tracks how many requests have been fired off and return true when that’s equal to the size of the batch? And if that’s the case I suppose synced would have similar logic I don’t think I need to do any work in prepare-batch right now, though

Yes, that’s exactly right.

You can decide whether you want to backpressure until the requests finish early in write-batch or just at the end with synced, depending on your needs.

With write-batch you can also stage your “unfired” requests in an atom, and continue to fire them off from write-batch, returning false until they’re done. It’s probably safer than counting them.

Yeah that’s a fair point

Is this writing to a particular DB/queue type product, or is it calling a web service? I’m just gathering info on what people are doing with the plugins.

It’s for invoking (or calling other methods on) AWS Lambdas. We had issues where tasks were failing to heartbeat due to long-running email requests that would error & timeout eventually (blocking on the call), which would cause retries and result in duplicate messages being sent. In this case we were also sending SMS messages in the same task, which would succeed, so we would get duplicated SMS messages

Ah, cool. This is a good answer to that problem. It’s much better than just increasing the subscriber and publisher timeouts.

Cool! Yeah we were hoping it would be 🙂