Hi! Where can I see example of transducer with internal state and cleanup? I am using async/pipeline fn with transducer to take page screenshots and want to cleanup browser windows. the code is https://gist.github.com/edbond/03f21089faf3e7923574

@edbond

Have you looked at the source code of the Clojure core functions that are now tranducers - some are stateful.

Thanks @meow, I figured out it's not possible to use stateful transducers with pipeline.

see https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/InsideTransducers/01.00.31.jpg

you can also maintain state via loop / recur

yes, I ended starting N go-loops with browser instance

So I've got a function, poll-source-and-put, which launches a go block, which is a background polling process that polls an external service and puts onto a channel. I've been wondering (and chatting here, with @bbloom a few days ago) about patterns for signaling that I want the process to terminate. I'd love to get a quick opinion here on four possible designs I've identified:

1. Consumer close!s the channel, to tell the polling process to stop. The argument against this is that channels are meant to represent one-way communication, and therefor only the putter should close a channel, and this violates the essence of the design. I find this argument convincing.

2. poll-source-and-put returns an atom to a flag which is checked in every iteration of the polling process. Just set the flag to false to stop polling, and then the producer can also cleanup the channel if desired.

3. poll-source-and-put returns a second "control channel". To tell the process to stop, you just close that control channel. This is elegant in a "channels" all the way down but feels slightly gratuitous to me.

i wouldn’t do #2, since you may wish to later switch from polling some thing to reading from a channel. in which case you’d need a timeout to poll the atom

4. poll-source-andput wraps up the details of either (2) or (3) and just returns a callable. You call it, and that stops the process (and cleans up if desired).

if you expose a channel instead of an atom, you can alt! instead of poll

+1 for option 3

definitely #3

why does it feel gratuitous?

Probably only b/c I'm new to channels. So I've seen flags in loops a million times in my life, and I haven't see alt! checking control channels in loops a million times. That may be the only reason, really. Not sure.

It's a feeling. (shrug)

there is nothing gratuitous - it doesn’t cost anything to poll two channels and you have a condp on the channel to keep the code paths cleanly separated

Yeah, I think it feels complex just because of conceptual overhead not computational overhead. Which is another way of saying just b/c it's unfamiliar to me. Which is why I'm curious of others' opinions.

(defn finite-printer [termination-ch data-ch]
  (let []
    (go-loop []
             (if-let [[data chan] (alts! [termination-ch data-ch])]
               (condp = chan
                 termination-ch (println data)              ; could check for val but, meh
                 data-ch (do
                           (clojure.pprint/pprint data)
                           (recur)))))))

trivial example

hmm. nice. And you reckon it's not a burden on the client of this code to have to create and provide the termination-ch ? Even tho it will only ever be used to control this one process? (Or I suppose you could ahve many processes watching one control channel... hmm.)

either way … less magic the better

@bbloom "you'd need a timeout to poll the atom". This is interesting to me. So you're pointing out one fundamental difference between an atomic flag and a control channel, is that a control channel is automatically synchronized with the rest of the process's channel processing. But a flag is checked whenever it is checked in the process's loop, so you need to start worrying about timing of "channel check" vs "flag check". And then that timing issue could be complicated if you have pauses for polling etc. Am I understanding right?

@alexisgallagher: you can place a watch on an atom

i think so, but the bigger issue is that if you change an implementation detail of your loop (eg switch from polling to blocked multiplexing) then you’d have to change your API from atom to channel to take advantage of the improvement from polling to blocking

so might as well start with a channel and keep your options open

but I agree with @bbloom that checking for an atom is more complex and as you say, channels can serve many processes

So there's a fundamental divide between polling and blocked multiplexing (as you put it). And one advantage of channels is that they can wrap both approaches. Yes, that makes sense.

yeah, polling is never free & if you can help it, you should avoid it

you need to tweak timeouts etc

your CPU keeps busy

it’s never ideal

Thanks, gentlemen. This has been educational.

I’m also thinking add-watch is also based on Thread so limits scaling