core-async 2023-09-12 | Slack Archive

I'm reducing over a collection, and I need to take from a channel inside of the reducing function.

(reduce (fn [acc item]
          (update acc :x conj (<! (get-some-async-stuff acc item)))))
        {}
        items)

This doesn't work because I need to be in a go-block and that means the accumulator needs to be a channel and everything gets messy very quickly. Is there a recipe for this sort of a thing, or is it just messy? I've started trying to implement via pipeline-async or maybe using transducers, but if this is a familiar problem I'd welcome some advice.

Jan K15:09:35

You can always turn it into a loop (or go-loop )

vlaaad15:09:53

https://github.com/defold/defold/blob/dev/editor/src/clj/editor/lsp/async.clj#L25-L44

hiredman15:09:34

Turn it into a map instead of a reduce, use pipeline for the map, use async/into to turn the result of the pipeline back into a collection

winsome17:09:08

The problem was a little more complex, it's a series of nested reduces for traversing arbitrarily-shaped trees. I ended up rewriting it as go-loops, which is less elegant but does the job.

hiredman17:09:12

reduce is particularly problematic because it is sort of its own callback based system (which is what CollReduce is) and that conflicts with the callback system that core.async is

winsome17:09:06

Yeah, I really wanted to try it as a transducer on a chan, but I wasn't able to make it work and loop was just sitting there the whole time.

hiredman17:09:31

I have played around a little bit with a version of iteration that iterates a process (a thing that runs over time on some channels) instead of a function, and I wonder what that would look like for reduce

hiredman17:09:32

(the reduce-async above is fine, but doesn't scratch that itch because af is a function that returns a channel, not a process)

winsome17:09:18

I've been playing around with Promesa's csp namespace with virtual threads and I think for green field work in the future I'll be skipping core.async - it's so much easier to just block and have the runtime do the right thing. Virtual threads are really great!

hiredman17:09:44

core.async's blocking ops should work just fine with virtual threads

winsome17:09:35

I think they do, but I like Promesa's api more.

hiredman17:09:52

I guess, promesa's csp too me looks like a rather mechanical copying of parts of core.async's api, without fixing any of the internal issues in core.async(particularly the resource handling around alts), using virtual threads, which are a brand spanking new feature.

winsome18:09:04

I think that's a pretty on-the-nose description. I do like that go just spawns a vthread instead of doing the whole state-machine macro dance, and all the limitations that brings, but obviously core.async couldn't do that because vthreads are new. And I also like the core promise library so using csp is just an extra bonus.

ghadi19:09:23

~~clojure.core.async/reduce exists.~~ different use case

hiredman21:09:14

https://downey.family/p/2023-09-12/async-reduce.clj.html maybe something like this is the higher order async-reduce, not sure anyone would actually want it ever

hiredman21:09:23

like a copy with a feedback loop

2023-09-12

Channels