Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-02-22

Channels

#core-async
<
2022-02-22
>
Ferdinand Beyer08:02:59

I stumbled over an implementation detail in core.async that puzzles me. We have pipeline and pipeline-blocking, with the latter being documented as “Like pipeline, for blocking operations.“. However, they seem to have exactly the same implementation: • Both delegate to pipeline*, passing a type argument with :compute for pipeline and :blocking for pipeline-blocking • In pipeline*, :compute and :blocking are treated the same: (case type (:blocking :compute) … :async …) Has that been changed since the original intention? Or is that to allow future optimizations?

Al Z. Heymer08:02:14

As far as I remember, pipeline will execute the xform in go -blocks, whilst pipeline-blocking executes it in thread -blocks

Ferdinand Beyer08:02:44

I thought so as well, but it does not. Both pipeline and pipeline-blocking will spin off threads to do the work: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L549

Al Z. Heymer08:02:52

Huh... Checked again, and it seems the impl I used to see is gone.

Ferdinand Beyer08:02:50

> • Release 0.5.527 on 2019.11.12 > ◦ … > ◦ Fix use of blocking op and thread constraints in `pipeline` - will now match `pipeline-blocking` in using N cached threads.

Ferdinand Beyer09:02:21

OK, there seems to have been a good reason to do so 😉

Ben Sless10:02:59

Using go-blocks for CPU is a bad idea so pipeline fixed that oversight. Theoretically, you can't have more threads running than CPU cores, therefor it makes sense for CPU threads to be used in a pool the size of available processors. Blocking IO, on the other hand, can take as many threads as you have memory available

Ben Sless10:02:27

Also theoretically, it's enough to have one thread for non blocking operations

Alex Miller (Clojure team)12:02:48

These impls may diverge again in the future