Fork me on GitHub

so, I've been moving some of my design around based on the stateful transducer issue talked about above Does the following seem like a reasonable example? I've often got a large-ish collection of compressed data files (represented by the take 20 range here that I want to do some series of mapping, grouping, reducing, mapping again. Is this a reasonable set up?

(let [inc-chan    (a/chan 32 (map inc))
        reduce-eo   (a/reduce (fn [acc n] (if (even? n)
                                            (update acc :even conj n)
                                            (update acc :odd conj n)))
                              {:even []
                               :odd  []}
        mapcat-chan (a/chan 8 (comp
                               (mapcat (fn [m] (map identity m)))
                               (mapcat (fn [[num-type nums]]
                                         (println "Numtype: " num-type " Nums: " nums)
                                         (into []
                                               (map (fn [n] [num-type n]))
                               (map (fn [[numtype n]] [numtype (inc n)]))))
        _           (a/pipe reduce-eo mapcat-chan)
        last-reduce (a/reduce (fn [acc [numtype n]]
                                (println "Numtype: " numtype " Num: " n)
                                (if (= :odd numtype)
                                  (update acc :used-to-be-odd conj n)
                                  (update acc :used-to-be-even conj n)))
                              {:used-to-be-even []
                               :used-to-be-odd  []}
    ;; Fire in events
    (a/onto-chan inc-chan (take 20 (range)))
    ;; collect events
    (a/<!! last-reduce))


I like using the higher order core.async function rather than doing something in a go block, unless I need to keep a look up table up to date or similar


this pattern of setting up the "machinery" in a let, followed by firing in the events, followed by retrieving the results from the channels is a pattern I'm following as well that seems to be working for me, but I don't know if there is a better way for this kind of problem


I'm mostly using this for embarrassingly parallel problems so the pipe would be replaced with some kind of pipeline for a lot of things

Alex Miller (Clojure team)16:10:25

is this different than a/pipeline?


the overall thing? or just the use of a/pipe?


I didn't think I could have a reducing stage in the middle of a/pipeline


this is a toy example of the kind of data processing I'd like to do


read in the records from a file, do scrubbing on each record, gather them up by some key (user id say), then process each data item grouped by a key and grab the results to put into some kind of output (csv, excel, database, api)


I've not found good examples of people chaining together the higher level core.async things (my google-fu has failed me)


and I'm just wondering if I'm using the library in a way it wasn't really mean to be

Alex Miller (Clojure team)16:10:19

Clojure Applied has a chapter on this but I think the key is - make components that are given the channels to use

Alex Miller (Clojure team)16:10:11

then whatever is assembling the units can make channels and hand it as an input to one and an output to another (and you don't even need pipe then)

Alex Miller (Clojure team)16:10:58

it also lets the assembler choose buffer policies and sizes etc


yeah, I think the pipe in here is a bit gratuitous looking at it again. That would probably be where I'd put in a a/pipeline if that was appropriate


I'll have a look again at the Clojure Applied chapter


re-reading it with that in mind should help

Alex Miller (Clojure team)16:10:10

I think it's ch 6 but I don't have it in front of me


I'm looking about for my dead tree version. I might have to go to safari

Alex Miller (Clojure team)16:10:50

I assume you mean Programming Clojure ;)


I've only got one copy of that


I think I gave away my dead tree copy of Clojure Applied (unless it is in another room). I push it at people who want to go from noodling at the repl to actually building things


it is the best book for that


the authors of all of them are really friendly


mmm... vertical slices


sounds then that things like a/mult should mostly live in the assembler namespaces


If I had a component doing a reduce/transduce/into then the component would have a pipe at the end to put the data on the passed in output channel I suppose (as those functions don't take output channels, but just return them


@alexmiller thx for this. This makes me think that a little assembly function I wrote for some sugar is almost completely a bad idea


I was thinking that was true before, but didn't have a good way of thinking how things should be assembled