Fork me on GitHub

@alexmiller I was wondering, why is pipeline not creating/blocking a thread later in the process, right before running a task. Right now it will create/block-on n threads even if nothing happens. I think that basically creates higher memory usage for nothing.


it could be like this instead no? (edited to latest version)


that one would cost more in context switches potentially, but at least, when the input chan is empty, 0 thread usage and low memory


I guess that patch could create some more work if the chan always contains values (async/thread call, since it's from a cached thread pool), but it's probably negligible given the usage context

Alex Miller (Clojure team)14:01:23

I think with your change, it's no longer bounded by n at all


There would never be more than n process calls parallel.


Since we take the result before recur


Actually no, you re right I think


Need less kids around to be able to play in repl, you put some doubt. But I think since every loop parks on process, every loop never runs more than one task at a time. At least it was the idea


not to mention I based that on a 6yo version of master....


a quick unscientific sprinkling of prn statements over an atom seems to show it's indeed limited to n


@alexmiller should I create a Jira with proper patch & test?

Alex Miller (Clojure team)19:01:16

what problem are we trying to solve here?


Yes, if you don't have n+ items to consume it will use less threads, potentially 0 if you are waiting on values to be sent to the chan, or when you are dealing with bursty loads.

Alex Miller (Clojure team)19:01:23

at the cost of spinning up those threads when needed


It doesn't change behavior if your chan producers are ahead of the pipeline "runner"


But you never have more than n threads running tasks at a time

Alex Miller (Clojure team)19:01:16

you can file something. to really evaluate I would need to dig out the original design stuff for this

Alex Miller (Clojure team)19:01:31

I don't think that's going to happen soon


It's possible to emulate this pattern with pipeline-async + thread on the current version. It's a bit ugly but it saves resources when idling


Alright, will do

Alex Miller (Clojure team)19:01:39

well, I'd lead with the problem


Lastly, since iirc we have a cached thread pool behind thread the price to "acquire" new threads is low, there's a lot of reuse going on

Alex Miller (Clojure team)19:01:21

cached thread pool threads die, so hard to say


They remain for 60s, it's quite a lot


That is what i remember, i could be wrong