Fork me on GitHub

What’s the state of the art for tuning a pipeline (in the generic sense)? For example, with two processes are parallelized using async/pipeline and friends, what tools/techniques do folks use to determine how big the channel buffer should be, the number of threads to parallelize for each, etc? I’m curious about making sure that threads from either pipeline call aren’t just sitting around.


Put another way, how can I get visibility into the “backpressure” that my system may be exerting?


if you are using pipeline (not pipeline blocking) there aren't any threads sitting around


pipeline runs things in go loops, so they reused the same fixed size threadpool that runs go loops, and that threadpool is limited to 8 threads by default


So I’m using pipeline-blocking in this particular case.


But that certainly makes sense in the go loop case


I wrote a thing that includes some examination of the flavors of pipelines the naive modeling equations are not correct (the data in the tables don't match them)


Definitely some interesting stuff in there.