What’s the state of the art for tuning a pipeline (in the generic sense)? For example, with two processes are parallelized using async/pipeline and friends, what tools/techniques do folks use to determine how big the channel buffer should be, the number of threads to parallelize for each, etc? I’m curious about making sure that threads from either pipeline call aren’t just sitting around.

Put another way, how can I get visibility into the “backpressure” that my system may be exerting?

if you are using pipeline (not pipeline blocking) there aren't any threads sitting around

pipeline runs things in go loops, so they reused the same fixed size threadpool that runs go loops, and that threadpool is limited to 8 threads by default

So I’m using pipeline-blocking in this particular case.

But that certainly makes sense in the go loop case

I wrote a thing that includes some examination of the flavors of pipelines https://ce2144dc-f7c9-4f54-8fb6-7321a4c318db.s3.amazonaws.com/buffer.html the naive modeling equations are not correct (the data in the tables don't match them)

Definitely some interesting stuff in there.