This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2015-07-31
Channels
- # admin-announcements (23)
- # announcements (3)
- # beginners (59)
- # boot (122)
- # clojure (117)
- # clojure-berlin (3)
- # clojure-dev (3)
- # clojure-italy (3)
- # clojure-japan (10)
- # clojure-russia (69)
- # clojurebridge (12)
- # clojurescript (183)
- # clojutre (14)
- # code-reviews (4)
- # core-async (79)
- # core-logic (18)
- # core-matrix (1)
- # cursive (7)
- # datascript (1)
- # datomic (18)
- # editors (3)
- # events (18)
- # hoplon (20)
- # ldnclj (24)
- # luminus (1)
- # off-topic (1)
- # other-lisps (1)
- # re-frame (12)
- # reagent (104)
@meow: This might sound harsh but: I wouldn't use/create a function like chan-of-primes
(except a convenience function) since it's very inflexible and can barely be re-used. All you can ever do is create a channel of primes. You instead want to transform data independently, ie. without information about channels. (if that's possible). If you go the transducer way you can still compose it with other transducers, apply it to your normal collections and to your channels. You should also accept input instead of hard-coding the chan-of-ints
. This means you've solved a more general problem which might allow future re-use of some kind.
Remember, in clojure it's all about functions. If your function doesn't accept any parameter then you've likely written a non-reusable function.
@rauh: I haven't even read what you wrote yet. I woke up feeling like I understood what was going on with the transducer you showed me yesterday so I dove into the code and, wow! Check it out:
(defn chan-of-primes []
(let [ints (chan-of-ints 2)
sieve (posmod-sift)
primes (chan 1 sieve)]
(pipe ints primes)
primes))
The full code is at the bottom of the wiki page: https://github.com/clojure/core.async/wiki/Sieve-of-Eratosthenes
@rauh: I agree with everything you've said and am ever so grateful for you being willing to be "harsh" because that's the kind of feedback I need to improve as a programmer (and you weren't really harsh at all)
After I worked up this new code I couldn't believe how ugly my previous attempts looked.
I just knew there had to be better ways to solve this problem and I wasn't seeing it in the wild.
And, yes, now that I got the version using only channels, I'll follow up with examples of how this transducer will let you work with other types of collections and contexts and how the channel-of-ints isn't even necessary.
@rauh: This is so cool - this final version of the code is the first one that works well in my cljs app - I'm able to run it and still get 60 frames per second.
@meow: Awesome! That's probably because of the volatile and mutable state. It's pretty fast.
at 30,000 primes found I'm down to 50fps, but that's not bad for a simple algorithm like SofE
I'm working on using things like core.async and goog.async and my own event loops to be able to have cljs apps that can sustain 60fps
I think this quote from Rich Hickey sums things up nicely:
In working recently on providing algorithmic combinators for core.async, I became more and more convinced of the superiority of reducing function transformers over channel->channel functions for algorithmic transformation.
The problem I have with that is that in my own experience, the majority of channel->channel functions aren’t algorithmic transformations but rather involve I/O… given this channel of entities, look up some other entities for each one and do X with them, etc.
Obviously that’s a function of the kind of systems that I’ve been working on, and not applicable to everyone, but I’m sure there’s a large subset of Clojure developers to whom it is.
@erik_price: From what I'm now understanding about transducers the nature of the algorithmic transformation is pretty open-ended.
Well, we might need to work up a specific example, but let's start with the premise that the operations you are describing could be coded as transducers.
Imagine I have a source of messages (coming off a distributed message queue). Each message goes into a channel pipeline, where they are first validated, then transformed – those can be done via transducers – but the next step of my channel pipeline is to fetch a file off of S3 and also issue a database query. My channel pipeline then joins this data into some data structure, and passes it along to the next phase. That step requires that I either use something like clojure.core.async/pipeline-async
, or break the channel pipeline up by introducing a go
block that pulls values out of the channel, performs the work, and then puts them back into a channel and passes that along to the next part of the pipeline.
Earlier, I said “the problem I have with that”, and that was not a wise choice of words. What I should have said is “there are certain transformations for which transducers cannot or should not be used”.
@erik_price: gotcha
So I'll probably push them by using them where they shouldn't be used, just to learn from my own mistakes.
But I suppose that putting a go block inside a transducer is probably a warning sign that I've gone too far...
To my understanding, which is not expert, blocking operations of any kind should be avoided in transducers. Just use them for, as Rich puts it, “algorithmic transformation”.
Keep in mind channel transducers run in the context of a channel mutex, so they may possibly exacerbate contention
@ghadi: are you saying that the execution of a transducer places an exclusive lock on the channel as far as take and put operations are concerned?
@ghadi: I was actually wondering how channel transducers are processed because I was thinking that syntax like the following would be nice:
(into (chan) xform (range 100))
and then I thought that the xform on a channel must somehow sort of treat it that way already
@meow: just as transducers transform some sort of "stepping function", channel transducers transform the add! operation of adding to a channel's buffer. https://github.com/clojure/core.async/blob/d07389619/src/main/clojure/clojure/core/async/impl/channels.clj#L288
@ghadi: so would there be any advantage to being able to treat a chan as a coll where something like into
just did a put!
onto the channel?
not sure that's what I'm looking for, but I find the docstring utterly confusing even though the code is only this:
(reduce conj coll ch)
what I was thinking of was a way to take a stream of values, like from (range) and use them as input into a chan
@rauh: just for the record, my earlier stats about frames per second while generating primes in the browser was with unoptimized code compilation - compiling with :advanced is way faster still
since I guess most of the time it's really early and only every now and then it has to run through the whole thing
not sure if you noticed that I have the transducer state using a vector instead of a set for that very reason - so every? will fail asap
Regarding into
: It has different semantics. The first argument is a collection which always allows adding elements. With channels you don't have that. You can say "channel full, hold on"
So it wouldn't make sense (or it would surprise people) to do extend that. Being explicit (thus other function) is better
from both ends, producer and consumer. Which is very powerful. (much more than other async
libraries in other languages)
No, the whole point about core.async is to invent synchronization points in your program so you can play with a new toy... 😉
so this is the final version I'm playing with now, and I think I'm quickly losing interest in taking this primes thing any further:
(defn chan-of-primes []
(let [inputs (filter odd? (drop 3 (range)))
primes (chan 1 (posmod-sift))]
(put! primes 2)
(onto-chan primes inputs)
primes))
Yeah, I'd be content with that solution. Not sure what more to poke around. I'm kinda out of ideas
well, I was toying with trying to force range to go from 3 to infinity with a step of 2 as a possible optimization but it would have to work on both clj and cljs and at some point this isn't really about optimizing the finding of primes anyhow but just having a good example
That's actually interesting: Write a pevery?
or peach?
that does runs with k
parallelism and each thread testing every k
th value.
Then coordinating an short circuit if one of them fails so the condition can't be satisfied