Fork me on GitHub
#core-async
<
2022-01-04
>
Ian Fernandez19:01:39

there's a way to have a channel that has java.lang.AutoCloseable implemented?

ghadi19:01:10

Just close it when you're done 🌈

ghadi19:01:47

If you're wanting autocloseable, you have control over lexical scope already

Ian Fernandez19:01:04

but It would be good to use channels in a system using https://github.com/piotr-yuxuan/closeable-map

ghadi19:01:48

that don't make no sense

ghadi19:01:32

a lot of times channels outlive lexical scope

Ian Fernandez20:01:06

It doesn't make sense to not bound channels to an outside system configuration and make handlers channel dependant that is in your system-map

Ian Fernandez20:01:28

to get rid of global unmanageable configurations

Ian Fernandez20:01:15

btw, it's very sad that "channels outlive lexical scope"

Ben Sless20:01:34

Every function which returns a channel or go block returns a channel which outlives its lexical scope

Ben Sless20:01:25

You can return channels as promises or flows, then they "escape" your scope

Ian Fernandez20:01:07

but, if I got a track on a system map for the chans that I've started and try to guarantee that they're close everytime I restart a system

Ben Sless20:01:51

Extend via metadata?

Ian Fernandez20:01:19

seems reasonable

Ben Sless20:01:31

But you'll also find the order becomes confusing

Ben Sless20:01:55

For clean shut down, the producer has to finish, then you close the channel, then the consumers

paulocuneo20:01:09

what happens if you close a channel before all its items are consumed?

Ian Fernandez20:01:30

I need to think about close a channel giving me some guarantees on every stuff being consumed.

paulocuneo20:01:26

I'm not an expert, but distributed systems shutdown is not a trivial problem. if you plan to do something like this, I guess you should: 1. stop the webserver traffic: either reject/redirect request or deactive instance network traffic. 2. send the stop signal, i.e. close the main channel and wait for all threads to finish 3. as @ben.sless said as each consumer receives the eof signal, it forwards the signal down the pipeline DAG . some interesting questions: what happens if you get duplicated eof? do you need to coordinate a consumer group/level shutdown? what happens if eof if forwarded to next level before sibling consumers are done? 4. when the DAG is done, stop/quit the process

👍 1
Ian Fernandez21:01:53

I agree with this for sure, but: 1. core.async does not seems a tool for this (???) 2. the system that I'm working doesn't have a way to shutdown in a way that I can build test scenarios, lot stuff coupled with these core.async shutdown triggers, etc.

Ben Sless20:01:24

If you build some dataflow I usually let the shutdown be implied by the topology

Ben Sless20:01:44

The risk is stopping the consumer before you drain the channel

Ben Sless20:01:00

When building pipelines I go by this principle - clean up after yourself. Producers are responsible for their channels. When they are done they close them, thus signalling to downstream consumers to shut down

Ian Fernandez20:01:33

I guarantee that my producer is finished

Ben Sless20:01:38

When consumers take nil from a channel it is closed and can initiate their own shutdown

Ben Sless20:01:11

I'll consider making the entire pipeline a single component

Ian Fernandez20:01:50

I'll add in the end in the system a kw like this ::closeable-map/after-close (fn [{::keys [my-chan]}] (a/close! mychan))

Ian Fernandez20:01:11

then I think it'll handle 🙂

Ian Fernandez20:01:27

because the producers are in the "other modules"

Ben Sless20:01:54

You have multiple producers to a single channel?

Ian Fernandez20:01:38

yes, but they live in a webserver that I stop before the channels

Ian Fernandez20:01:30

I need to think about close a channel giving me some guarantees on every stuff being consumed.

Ben Sless20:01:38

> Multiple producers in a web server

Ben Sless20:01:55

Please tell me you're using a dropping or sliding buffer at least?

Ian Fernandez20:01:30

why it should matter? :thinking_face:

Ian Fernandez20:01:43

sorry, kinda newbie here 😅

Ben Sless20:01:25

It's getting late here so I'd rather continue tomorrow but tldr: blocking in a web handler is a recipe for trouble

Ian Fernandez20:01:26

which would be the length of my channel buffer? a number of webserver connections limit?

Ben Sless20:01:45

No, the channel you're putting on

Ian Fernandez20:01:52

got wrong question, I've edited it

Ben Sless20:01:25

In steady state, a channel is either full or empty. You only need a buffer to handle spikes

Ben Sless20:01:22

What you need is a way to send 400 when you're flooded

Ben Sless20:01:53

But channels don't give you feedback, and I guess you don't produce with alts and timeout

Alex Miller (Clojure team)20:01:15

offer! can give you feedback

1
paulocuneo21:01:07

Just out of curiosity, from previous conversation... Is it posible to have an "eof race condition" when closing a channel to multiple consumers? By "eof race condition" I mean that one go block/threads see the eof before the other sibling consumers, an so the "first" to see the eof would forward the eof before the siblings are done?

Alex Miller (Clojure team)21:01:06

that's not how it works

Alex Miller (Clojure team)21:01:39

channels are stateful, and state changes occur under a lock

Alex Miller (Clojure team)21:01:01

so there is no visibility race

Alex Miller (Clojure team)21:01:18

if two consumers go to consume, that is a race of course in the way of any concurrent system and only one will "consume" a value. but eof is not a value consumers race to consume, but a condition triggered by a flag that consumers will observe (by receiving nil on consume)

Ben Sless21:01:55

So don't invent unique stop signals 🙂

paulocuneo21:01:01

thanks @alexmiller, great answer, I didn't mean inside core.async ,I meant if you design/build a DAG with consumer groups, (maybe not the best application for core.async). I guess from your answer that a complex DAG requires a coordinated shutdown anyway. @ben.sless yeah, maybe avoid complex DAG at all

Alex Miller (Clojure team)21:01:36

I mean, you don't necessarily have to close at all

Alex Miller (Clojure team)21:01:26

and you could give processes an independent shutdown channel too, separate from data flow

Alex Miller (Clojure team)21:01:47

have a large helping of "it depends"

👍 1