This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-10-23
Channels
- # announcements (1)
- # architecture (20)
- # babashka (30)
- # beginners (79)
- # calva (27)
- # cider (8)
- # clj-kondo (1)
- # clojure (125)
- # clojure-australia (1)
- # clojure-berlin (4)
- # clojure-europe (62)
- # clojure-france (1)
- # clojure-italy (6)
- # clojure-nl (4)
- # clojure-uk (12)
- # clojuredesign-podcast (5)
- # clojurescript (28)
- # core-async (31)
- # cursive (14)
- # datomic (47)
- # defnpodcast (1)
- # emacs (7)
- # figwheel-main (2)
- # fulcro (10)
- # graalvm (1)
- # graphql (11)
- # jobs-discuss (8)
- # leiningen (2)
- # off-topic (23)
- # rdf (9)
- # re-frame (3)
- # reagent (1)
- # reitit (5)
- # reveal (12)
- # shadow-cljs (12)
- # spacemacs (1)
- # tools-deps (87)
- # vim (22)
- # xtdb (21)
so, I've been moving some of my design around based on the stateful transducer issue talked about above https://clojurians.slack.com/archives/C05423W6H/p1603202921084800 Does the following seem like a reasonable example? I've often got a large-ish collection of compressed data files (represented by the take 20 range here that I want to do some series of mapping, grouping, reducing, mapping again. Is this a reasonable set up?
(let [inc-chan (a/chan 32 (map inc))
reduce-eo (a/reduce (fn [acc n] (if (even? n)
(update acc :even conj n)
(update acc :odd conj n)))
{:even []
:odd []}
inc-chan)
mapcat-chan (a/chan 8 (comp
(mapcat (fn [m] (map identity m)))
(mapcat (fn [[num-type nums]]
(println "Numtype: " num-type " Nums: " nums)
(into []
(map (fn [n] [num-type n]))
nums)))
(map (fn [[numtype n]] [numtype (inc n)]))))
_ (a/pipe reduce-eo mapcat-chan)
last-reduce (a/reduce (fn [acc [numtype n]]
(println "Numtype: " numtype " Num: " n)
(if (= :odd numtype)
(update acc :used-to-be-odd conj n)
(update acc :used-to-be-even conj n)))
{:used-to-be-even []
:used-to-be-odd []}
mapcat-chan)]
;; Fire in events
(a/onto-chan inc-chan (take 20 (range)))
;; collect events
(a/<!! last-reduce))
I like using the higher order core.async function rather than doing something in a go block, unless I need to keep a look up table up to date or similar
this pattern of setting up the "machinery" in a let, followed by firing in the events, followed by retrieving the results from the channels is a pattern I'm following as well that seems to be working for me, but I don't know if there is a better way for this kind of problem
I'm mostly using this for embarrassingly parallel problems so the pipe would be replaced with some kind of pipeline for a lot of things
is this different than a/pipeline?
read in the records from a file, do scrubbing on each record, gather them up by some key (user id say), then process each data item grouped by a key and grab the results to put into some kind of output (csv, excel, database, api)
I've not found good examples of people chaining together the higher level core.async things (my google-fu has failed me)
Clojure Applied has a chapter on this but I think the key is - make components that are given the channels to use
then whatever is assembling the units can make channels and hand it as an input to one and an output to another (and you don't even need pipe then)
it also lets the assembler choose buffer policies and sizes etc
yeah, I think the pipe in here is a bit gratuitous looking at it again. That would probably be where I'd put in a a/pipeline if that was appropriate
I think it's ch 6 but I don't have it in front of me
I assume you mean Programming Clojure ;)
I think I gave away my dead tree copy of Clojure Applied (unless it is in another room). I push it at people who want to go from noodling at the repl to actually building things
If I had a component doing a reduce/transduce/into then the component would have a pipe at the end to put the data on the passed in output channel I suppose (as those functions don't take output channels, but just return them
@alexmiller thx for this. This makes me think that a little assembly function I wrote for some sugar is almost completely a bad idea