Fork me on GitHub

We have designed the output plugin interfaces to allow you to implement async requests like this in a sensible way.


You can see it in play in the sqs plugin.


prepare-batch allows you to get all of your requests together


then write-batch allows you to initiate your requests, and return false if you’re not done, or true if you are done


Generally I’d recommend returning true if you’ve started all of your requests, or false if you have only started some of them and you want to stay at the write-batch stage


Then the additional missing piece is that you HAVE to ensure all of your requests have completed in the synced? call


The sync call is what allows a barrier to progress through, and if you have synced you are essentially signalling that you have completed everything between two barriers and it is safe to checkpoint.


The idea is that you have two places you can backpressure (write-batch and synced?). write-batch and prepare-batch also allow you to yield and not progress to the next state, which allows you to give back the thread to the peer so that it can heartbeat if your request is long running. That way your peer doesn’t get timed out.

Graham Seyffert00:04:52

So if write-batch returns false, the task will yield execution back to the thread? So basically I can do something like the http plugin that tracks how many requests have been fired off and return true when that’s equal to the size of the batch? And if that’s the case I suppose synced would have similar logic I don’t think I need to do any work in prepare-batch right now, though


Yes, that’s exactly right.


You can decide whether you want to backpressure until the requests finish early in write-batch or just at the end with synced, depending on your needs.


With write-batch you can also stage your “unfired” requests in an atom, and continue to fire them off from write-batch, returning false until they’re done. It’s probably safer than counting them.

Graham Seyffert00:04:20

Yeah that’s a fair point


Is this writing to a particular DB/queue type product, or is it calling a web service? I’m just gathering info on what people are doing with the plugins.

Graham Seyffert00:04:18

It’s for invoking (or calling other methods on) AWS Lambdas. We had issues where tasks were failing to heartbeat due to long-running email requests that would error & timeout eventually (blocking on the call), which would cause retries and result in duplicate messages being sent. In this case we were also sending SMS messages in the same task, which would succeed, so we would get duplicated SMS messages


Ah, cool. This is a good answer to that problem. It’s much better than just increasing the subscriber and publisher timeouts.

Graham Seyffert00:04:38

Cool! Yeah we were hoping it would be 🙂