Fork me on GitHub
#clojure-dev
<
2021-01-25
>
mpenet11:01:08

@alexmiller I was wondering, why is pipeline not creating/blocking a thread later in the process, right before running a task. Right now it will create/block-on n threads even if nothing happens. I think that basically creates higher memory usage for nothing.

mpenet11:01:28

it could be like this instead no? https://github.com/mpenet/core.async/commit/e9bb61d (edited to latest version)

mpenet11:01:02

that one would cost more in context switches potentially, but at least, when the input chan is empty, 0 thread usage and low memory

mpenet11:01:15

I guess that patch could create some more work if the chan always contains values (async/thread call, since it's from a cached thread pool), but it's probably negligible given the usage context

alexmiller14:01:23

I think with your change, it's no longer bounded by n at all

mpenet14:01:39

There would never be more than n process calls parallel.

mpenet14:01:57

Since we take the result before recur

mpenet14:01:51

Actually no, you re right I think

mpenet15:01:51

Need less kids around to be able to play in repl, you put some doubt. But I think since every loop parks on process, every loop never runs more than one task at a time. At least it was the idea

mpenet15:01:15

not to mention I based that on a 6yo version of master....

mpenet15:01:25

a quick unscientific sprinkling of prn statements over an atom seems to show it's indeed limited to n

mpenet19:01:08

@alexmiller should I create a Jira with proper patch & test?

alexmiller19:01:16

what problem are we trying to solve here?

alexmiller19:01:57

using fewer threads?

mpenet19:01:53

Yes, if you don't have n+ items to consume it will use less threads, potentially 0 if you are waiting on values to be sent to the chan, or when you are dealing with bursty loads.

alexmiller19:01:23

at the cost of spinning up those threads when needed

mpenet19:01:42

It doesn't change behavior if your chan producers are ahead of the pipeline "runner"

mpenet19:01:06

But you never have more than n threads running tasks at a time

alexmiller19:01:16

you can file something. to really evaluate I would need to dig out the original design stuff for this

alexmiller19:01:31

I don't think that's going to happen soon

mpenet19:01:57

It's possible to emulate this pattern with pipeline-async + thread on the current version. It's a bit ugly but it saves resources when idling

mpenet19:01:12

Alright, will do

alexmiller19:01:39

well, I'd lead with the problem

mpenet19:01:49

Lastly, since iirc we have a cached thread pool behind thread the price to "acquire" new threads is low, there's a lot of reuse going on

alexmiller19:01:21

cached thread pool threads die, so hard to say

mpenet19:01:17

They remain for 60s, it's quite a lot

mpenet19:01:15

That is what i remember, i could be wrong

alexmiller19:01:33

that's right