I have an image pipeline where some tasks (4 of them) are going to be CPU bound and some (1 of them) are going to be io bound (uploading to s3). I am a bit confused as to the effect of batching vs. number of peers.

Are there other variables I should be messing with to try to get the pipeline to run a bit faster?

@chrisn: You probably want a batch size of 1.

Performance tuning is described here: https://github.com/onyx-platform/onyx/blob/master/doc/user-guide/performance-tuning.md#onyx

There are descriptions of more knows that you can tune: https://github.com/onyx-platform/onyx/blob/master/doc/user-guide/peer-config.md

You're looking to tune :onyx/batch-size, :onyx-timeout, :onyx/max-pending, and :onyx/pending-timeout

On the run, let me know if you have any more questions