Fork me on GitHub
#core-async
<
2016-02-15
>
edbond15:02:17

Hi! Where can I see example of transducer with internal state and cleanup? I am using async/pipeline fn with transducer to take page screenshots and want to cleanup browser windows. the code is https://gist.github.com/edbond/03f21089faf3e7923574

meow17:02:42

Have you looked at the source code of the Clojure core functions that are now tranducers - some are stateful.

edbond18:02:50

Thanks @meow, I figured out it's not possible to use stateful transducers with pipeline.

genRaiy18:02:46

you can also maintain state via loop / recur

edbond18:02:55

yes, I ended starting N go-loops with browser instance

alexisgallagher18:02:16

So I've got a function, poll-source-and-put, which launches a go block, which is a background polling process that polls an external service and puts onto a channel. I've been wondering (and chatting here, with @bbloom a few days ago) about patterns for signaling that I want the process to terminate. I'd love to get a quick opinion here on four possible designs I've identified:

alexisgallagher18:02:02

1. Consumer close!s the channel, to tell the polling process to stop. The argument against this is that channels are meant to represent one-way communication, and therefor only the putter should close a channel, and this violates the essence of the design. I find this argument convincing.

alexisgallagher18:02:41

2. poll-source-and-put returns an atom to a flag which is checked in every iteration of the polling process. Just set the flag to false to stop polling, and then the producer can also cleanup the channel if desired.

alexisgallagher18:02:25

3. poll-source-and-put returns a second "control channel". To tell the process to stop, you just close that control channel. This is elegant in a "channels" all the way down but feels slightly gratuitous to me.

bbloom18:02:42

i wouldn’t do #2, since you may wish to later switch from polling some thing to reading from a channel. in which case you’d need a timeout to poll the atom

alexisgallagher18:02:52

4. poll-source-andput wraps up the details of either (2) or (3) and just returns a callable. You call it, and that stops the process (and cleans up if desired).

bbloom18:02:00

if you expose a channel instead of an atom, you can alt! instead of poll

genRaiy18:02:10

+1 for option 3

bbloom18:02:30

definitely #3

genRaiy18:02:46

why does it feel gratuitous?

alexisgallagher18:02:33

Probably only b/c I'm new to channels. So I've seen flags in loops a million times in my life, and I haven't see alt! checking control channels in loops a million times. That may be the only reason, really. Not sure.

alexisgallagher18:02:04

It's a feeling. (shrug)

genRaiy18:02:14

there is nothing gratuitous - it doesn’t cost anything to poll two channels and you have a condp on the channel to keep the code paths cleanly separated

alexisgallagher18:02:08

Yeah, I think it feels complex just because of conceptual overhead not computational overhead. Which is another way of saying just b/c it's unfamiliar to me. Which is why I'm curious of others' opinions.

genRaiy18:02:51

(defn finite-printer [termination-ch data-ch]
  (let []
    (go-loop []
             (if-let [[data chan] (alts! [termination-ch data-ch])]
               (condp = chan
                 termination-ch (println data)              ; could check for val but, meh
                 data-ch (do
                           (clojure.pprint/pprint data)
                           (recur)))))))

genRaiy18:02:55

trivial example

alexisgallagher18:02:59

hmm. nice. And you reckon it's not a burden on the client of this code to have to create and provide the termination-ch ? Even tho it will only ever be used to control this one process? (Or I suppose you could ahve many processes watching one control channel... hmm.)

genRaiy18:02:06

either way … less magic the better

alexisgallagher18:02:52

@bbloom "you'd need a timeout to poll the atom". This is interesting to me. So you're pointing out one fundamental difference between an atomic flag and a control channel, is that a control channel is automatically synchronized with the rest of the process's channel processing. But a flag is checked whenever it is checked in the process's loop, so you need to start worrying about timing of "channel check" vs "flag check". And then that timing issue could be complicated if you have pauses for polling etc. Am I understanding right?

genRaiy18:02:52

@alexisgallagher: you can place a watch on an atom

bbloom18:02:54

i think so, but the bigger issue is that if you change an implementation detail of your loop (eg switch from polling to blocked multiplexing) then you’d have to change your API from atom to channel to take advantage of the improvement from polling to blocking

bbloom18:02:25

so might as well start with a channel and keep your options open

genRaiy18:02:35

but I agree with @bbloom that checking for an atom is more complex and as you say, channels can serve many processes

alexisgallagher18:02:36

So there's a fundamental divide between polling and blocked multiplexing (as you put it). And one advantage of channels is that they can wrap both approaches. Yes, that makes sense.

bbloom18:02:58

yeah, polling is never free & if you can help it, you should avoid it

bbloom18:02:03

you need to tweak timeouts etc

bbloom18:02:07

your CPU keeps busy

bbloom18:02:09

it’s never ideal

alexisgallagher18:02:12

Thanks, gentlemen. This has been educational.

genRaiy18:02:27

I’m also thinking add-watch is also based on Thread so limits scaling