core-async 2021-09-24 | Slack Archive

Jakub Holý (HolyJak)08:09:12

Hello! I am trying to put together a list of "golden rules" of core.async. What do you think about this: General 1. Respect back-pressure, i.e. do not try to put stuff into a channel faster than it is consumed (beware of the “max 1024 pending puts”) 2. Have a strategy for error handling Go blocks 1. Never call a blocking operation within a go block (including <!!, >!!) 2. Go blocks stop at the function boundary, i.e. if you call a function using parking channel operations, it must have its own go block and return its channel to enable back-pressure ? Anything missing? Anything wrong? 🙏

bortexz08:09:10

do not try to put stuff into a channel faster than it is consumed

I would say, in this case, to use parking/blocking operations >! >!!, instead of put! . That will wait until the message is accepted by the channel. When different channels are connected on a pipeline using these operations in loops, you guarantee that the producer won’t be faster than the consumer, because it will always wait until messages can be consumed. That’s how you achieve backpressure. Avoid using put! or (go (>! …)) as these don’t wait for messages to be consumed, so then you can end up with faster producers than consumers

bortexz08:09:40

If you look at the implementations of pipe , mult , pub etc they all use >! in go-loops, so they will never put into channels faster than what the consumer (and buffers) can support

Jakub Holý (HolyJak)09:09:48

When I set put i did not mean (just) put! 🙂 It was exactly to cover what you said. Do you have an eloquent way to change my sentence to make it clearer while still keeping it short?

Ben Sless10:09:40

yes, this is more of a heuristic - try to not do anything "significant" in go blocks

Ben Sless10:09:51

the obvious - don't do blocking IO

Ben Sless10:09:23

the less obvious, don't do heavy CPU. What's the difference between doing blocking IO and heavy CPU operations? They both hog the thread

👍 1

Ben Sless10:09:17

Since the go-blocks thread pool is global and is not a fork-join pool, if you fill it up with cpu ops the threads will be too busy to transfer stuff between channels

Ben Sless10:09:05

Another heuristic I've adopted is separate logistics, wiring and business logic. Try to write business logic which is unaware of channels, and async functions which are unaware of business logic

❤️ 1

bortexz10:09:34

@U0522TWDA Maybe something along these? What do you think?

Respect backpressure, i.e don't do unbounded puts in channels, things like put! or (go (>! ...)) in loops

❤️ 1

bortexz10:09:40

Or to your original sentence, maybe I would add a small note like Use >! or >!! in loops instead of put! to adapt to consumers speed

👍 1

Jakub Holý (HolyJak)13:09:01

Great, thank you, Alberto!

didibus19:09:58

You could say parking put for >! and parking take for <!. Blocking put is >!!, and blocking take is <!!. And async put is put!, and async take is take!

👍 1

didibus19:09:27

And then put and take refers to one of the many puts or takes.

didibus19:09:50

Can someone explain the naming convention of core.async with regards to one ! or two !! ? At first I thought the single ! was reserved to things you must use inside a go block. But that's not true, for example for put! or poll!. So now all I'm seeing is that !! means that it is blocking. And that something ends with ! or not seems to be irrelevant, since it could be that you have to use it inside go or not.

Alex Miller (Clojure team)19:09:27

! is parking op, ok in go. !! is blocking, do not use in go

didibus20:09:41

Oh, so does ! indicate that it will use a go/park internally? Like I'm wondering for example with poll! ? Does that mean that poll! when used in a go block will park?

hiredman21:09:20

it is more like single ! implies it is ok to use in a go block

hiredman21:09:45

some single ! operations can only be used a go block

hiredman21:09:05

!! operations should never be used in a block

didibus23:09:28

Hum, I think some operations like into are also safe inside a go block. I guess, maybe !is a little irrelevant? Or inconsistent, but !!indicates blocking consistently.

2021-09-24

Channels