Fork me on GitHub
#admin-announcements
<
2015-09-03
>
tel00:09:23

teamaverage: there’s something to that… they move all of the work into function composition and keep the data structures out of it

tel00:09:17

there also ought to be a way to look at them through the lens of a CPS transform which will connect up with other structures which do work like “left associate all the binds"

teamaverage00:09:57

ha ha lost me : )

tel00:09:11

well, I’ll just say that you’re definitely on the right track simple_smile

teamaverage00:09:47

... and the form (map-t [rf] (fn [f] ... ) is not the reducer but the transducer?

teamaverage00:09:00

it lets you compose transducers?

teamaverage00:09:25

I think I'm getting caught out in what's a transducer and what's plumbing for composing them ...

tel00:09:26

if I say that a reducer is the shape (value, reduct -> reduct) does that fit with your intuition?

teamaverage00:09:57

Yeah, then transucer is (value, reduct -> reduct) -> (value, reduct -> reduct) ?

tel00:09:07

a “reducer transformer"

tel00:09:39

now I’m going to make a simplification

tel00:09:55

and alias (value, reduct -> reduct) to X

tel00:09:04

so that a transducer is just a normal looking function

tel00:09:27

now transducer composition piping should be more clear

tel00:09:34

since it’s identical to function composition

tel00:09:03

you just stick them end-to-end (X -> X) (X -> X) (X -> X) and glue the pieces together

teamaverage00:09:14

And that way you're talking about input and output values with no mention of collections

tel00:09:42

well, the collection vanishes because reducers just see one value at a time

tel00:09:53

they just “visit” values one at a time

teamaverage00:09:57

What does the "traversing" then? The final thing that takes you pass your composed functiuon into?

tel00:09:15

so far the “reduct” bit has been abstract

tel00:09:46

and furthermore, a transducer has to be “kicked off” by providing the first argument

tel00:09:52

which is a reducer itself

tel00:09:26

so if t : X -> X then we need to first do (t r) where r : (value, reduct -> reduct)

tel00:09:35

but we get to pick what r is

tel00:09:53

conj is a good choice simple_smile

tel00:09:18

rather, #(conj %2 %1)

tel00:09:17

when we do that we’ll get a reducer out (value, reduct -> reduct) which we can call reduce on to “consume” the input structure

teamaverage00:09:18

yeah so when you seed the t with r you get your final, composed reducer

tel00:09:45

and you’ll note that a reducer can be applied to any data structure which “has values” in some order

tel00:09:25

since you just feed an initial reduct in, which you need to get from somewhere, but it’ll be determined by your output… maybe it’s an empty vector, or an empty set

tel00:09:41

and then the process can be kicked off, eating values one after another

teamaverage00:09:23

Cause after it all, the final reducer still has the signature (value, reduct -> reduct) which (reduce ...) knows how to use ?

teamaverage00:09:41

Do you still use (reduce .. )?

tel00:09:08

in this basic form you do

tel00:09:39

there’s one more trick going on with the reduced function

tel00:09:53

which basically lets you exit early if your final reduction is smart enough to catch it

tel00:09:02

but conceptually you’re on the mark

teamaverage00:09:48

So I also read about reducers namespace which predates transducers are they solving different things or accomplishing the same things but with different approaches?

tel00:09:27

think it’s the same idea, transducers is just the next step of the evolution

teamaverage00:09:31

I can see the transducers as more reusable as you can just assign your final transduced value to something

teamaverage00:09:27

Getting away from all those implicit seq/coll that get created if you were to use the usual right -> left nesting

tel00:09:49

yeah, it lets you build actions together instead of operating them

tel00:09:56

which can be good and bad

tel00:09:06

if one step in your transduction chain is really expensive

tel00:09:16

you might actually want to produce that intermediate value

tel00:09:27

so you can cache it if you transduce repeatedly

tel00:09:43

but generally without the language of transducers you don’t even have that choice

tel00:09:48

so they’re nice in that way

teamaverage00:09:41

I really like how you can apply them to channels to automatically transform channel values

teamaverage00:09:14

Saves you having to read from the channel, transform, then put on another channel

tel00:09:18

you can think simply of them as just functions a -> [b] if you like

tel00:09:45

so if you apply it to a channel you read off an a value and write some number, 0 to many, b values back

teamaverage00:09:13

Yeahp very nice

meow00:09:41

I first got into tranducers via channels. I like how they abstract out the collection so it doesn't matter - very generic. Also that they can be composed and eliminate intermediate results, so they should perform nicely.

meow01:09:50

I'm still having to work at them so that they are more familiar and come naturally.

teamaverage01:09:04

@meow: "still having to work at them so that they are more familiar" here here. Though I'm also relatively new to Clojure

roberto01:09:37

I still don’t grok transducers. Maybe because most of the examples I’ve seen are too abstract that I can’t see how it can help.

meow03:09:56

@roberto: do you work with core.async channels at all? If so you might find this interesting:

(defn posmod-sift []
  (fn [xf]
    (let [seen (volatile! [])]
      (fn
        ([] (xf))
        ([result] (xf result))
        ([result input]
         (if (every? #(pos? (mod input %)) @seen)
           (do (vswap! seen conj input)
               (xf result input))
           result))))))

(defn chan-of-primes []
  (let [inputs (filter odd? (drop 3 (range)))
        primes (chan 1 (posmod-sift))]
    (put! primes 2)
    (onto-chan primes inputs)
    primes))

roberto03:09:37

no, I don’t do much in channels. Only in the front-end when doing xhr requests

roberto03:09:02

haven’t had any need for doing anything serious with core.async yet.

roberto03:09:39

i’ve only played with clojure in my personal projects. Don’t do anything with it in my day job.

meow03:09:44

I need to edit it. I used xf because that's what the official docs use but I prefer to call the reducing function rf

meow03:09:06

@roberto: ah, I hear ya

roberto03:09:23

yeah, maybe that is what confuses me when reading about transducers, the tendency to use single char vars

teamaverage03:09:22

@roberto: go through some of David Nolans core.async channel examples on his site - he makes great use of them. They're great as an abstraction over browser events e.g. mouse clicks and what not

roberto03:09:42

yeah, I saw his webinar at cognitect

roberto03:09:46

I’ll give that another read. Takes me a while to grok things, mostly only after i’ve found a real world use for it.

roberto03:09:59

after almost a year, I’m finally understanding clojure simple_smile

teamaverage03:09:35

Yeah same. I found http://kukuruku.co/hub/funcprog/clojure-transducers-reducers-and-other-stuff quite good for understanding transducers - especially where he explodes them into casual functions

meow03:09:26

One suggestion that was given to me, which helped a lot, is to look at the source code for some of the core functions that now return transducers, like map and partition-by.

roberto03:09:57

cool, didn’t know map returned a transducer

meow03:09:34

"The following functions produce a transducer when the input collection is omitted: map cat mapcat filter remove take take-while take-nth drop drop-while replace partition-by partition-all keep keep-indexed map-indexed distinct interpose dedupe random-sample"

meow03:09:29

So one way to play with transducers is to just use any of the core functions without specifying a coll and you get back a transducer. Then you can compose them together in different combinations. Then use something like transduce my-comp-of-transducers conj [1 2 3 4 5 whatever]

meow03:09:38

or into [] my-comp-xform some-coll

meow03:09:00

so much of the core is transducer capable

meow03:09:31

then if you need a custom transducer, like in the primes example I gave above, you can write a custom one

meow03:09:00

the power and complexity and unfamiliarity is due to the fact that they operate without regard for the underlying collection (don't know, don't care) and we aren't used to that in most other languages

meow03:09:00

transducers really are one heck of a good idea

teamaverage03:09:37

Is that why if you want a coll at the end, often the last comp of a transducer chain is conj?

teamaverage03:09:04

Cause usually those functions return collections don't they?

teamaverage03:09:04

Also, is transduce essentially used to unwrap the xform f from transducer to reducer and kick of the reduction?

meow03:09:48

@teamaverage: not sure I follow you

teamaverage03:09:34

So, (transduce xform f coll)

teamaverage03:09:50

the xform f will produce a transducer, right?

teamaverage03:09:55

Okay, problem solved

meow03:09:58

xform is the transducer

meow03:09:15

f is some reducing function, like conj

teamaverage03:09:40

will xform not be called with f to make the final reducter?

meow03:09:17

I'd have to check which gets called with what for the final step - I haven't had a need to worry about that part yet.

meow03:09:16

That completion step isn't needed very often.

meow03:09:55

Mostly if its a transducer that has state and after every item in the original collection has been processed the transducer needs to be called one final time to flush out whatever it is retaining in its state then that's the chance for it to do so.

meow03:09:55

So you can go a long way with transducers without having to worry about that part of it.

meow03:09:10

Transducers do have that magical quality to them where the actual process is hidden from view and, much like middleware, you just specify the things that you want to have happen, but the happening happens automagically. If that makes sense. If you are familiar with middleware or boot tasks it should make sense.

meow03:09:37

@tel: You mentioned early termination of transducers, and I haven't really worked with that aspect of them. Got any example of how that is useful?

teamaverage03:09:42

Yeah, I think I'm tripping up not being able to see traversal code

teamaverage03:09:55

Maybe this is my brain making the imperative -> functional leap ha ha

meow03:09:08

@teamaverage: Yeah, I think transducers really kick things up even one level higher so you've got more abstraction removing you from imperative code and then you have this sort of middleware-ish "inversion of control"ish aspect as well.

teamaverage04:09:30

Yeahp, they sure do

meow04:09:34

The payoff, especially the way they have been interwoven into the existing core code base, is that the same functions that you learn in clojure, like map, filter, partition-by, etc. can not only be used the regular way, but can also provide transducers. So you don't have to learn a whole different set of transducers. You just learn the concept of transducers and then use the transducer forms of all the regular core functions.

meow04:09:46

Imagine if you had to learn the concept of transducers and then a whole bunch of totally unfamiliar transducers with weird names - no way!

teamaverage04:09:27

Yeahp, they're tricky enough as it is ; )

meow04:09:24

Another way to look at transducers is just like the regular function, like say filter odd? or take 5 that we already know how to use by supplying them with a collection to operate on and instead were saying, hey, give me the ability to take 5 items and filter for the odd ones, but I'm not going to tell you what collection to do that to until later.

meow04:09:21

And then later, when I tell you the collection, I don't want you to care about the collection specifically, I just want to feed you items and something else will worry about those collection details.

teamaverage04:09:17

So the transducer-equivalents of map/filter and all that are collection ignorant?

meow04:09:54

And since you no longer care about the collection details, why don't we just comp you guys together so you can pass the item from one of you to the other instead of computing a bunch of intermediate results cuz that's just so like '90s inefficient and we're being really functional here.

teamaverage04:09:03

(just reading the core.map source now)

meow04:09:25

yeah, so check out map called without a collection:

([f]
    (fn [rf]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
           (rf result (f input)))
        ([result input & inputs]
           (rf result (apply f input inputs))))))

meow04:09:23

It returns a transducer

teamaverage04:09:32

Yip i see that

teamaverage04:09:19

Is there a way to print out functions in the REPL?

meow04:09:30

you mean to see the source code for a function?

meow04:09:07

I know there is cuz I used that to add that feature to devcards, but I never actually use the feature in the repl

meow04:09:28

I use cursive so I just use it to find the source

meow04:09:31

So it's late and I'm tired and the closure docs do a better job explaining it but all transducers have that weird but elegant shape of a function that returns a function that has 3 arities for the various parts of the transducing process.

teamaverage04:09:31

It's mid afternoon here in Australia. Thanks for your help

teamaverage04:09:30

I'm sure I'll get the lightbulb moment soon enough

teamaverage04:09:46

Probably just need to try write more transducer code myself instead of readin

meow04:09:36

What you want to do now is imagine what goes on when you use the transducer returned by map in a specific context, like this:

(transduce (map inc) conj [1 2 3 4])

meow04:09:40

would help if I typed it correctly

meow04:09:59

okay, so (map inc) gets us the transducer, which will be called by transduce

meow04:09:38

we don't supply an intial value so conj will be called and expected to provide one, which it does: []

meow04:09:54

that becomes result in calls to map

meow04:09:22

rf is conj, result is [], f is inc and input will be first 1 then in the next call 2, then 3, etc

meow04:09:30

So this is the main workhorse of the whole deal:

([result input]
           (rf result (f input)))

teamaverage04:09:47

Yeah, that's the reducer right

meow04:09:32

it gets called for each item in the original coll, which we made [1 2 3 4]

meow04:09:22

so that will look like (conj [] (inc 1)), then (conj [2] (inc 2)) then (conj [2 3] (inc 3))

meow04:09:59

cool, hope that helped

teamaverage04:09:06

Indeed, thanks : )

meow04:09:40

explaining it is helping solidify it in my own mind

teamaverage04:09:31

Ha ha you're probably getting even more out of it than me ; )

meow04:09:56

probably, I won't lie simple_smile

teamaverage04:09:18

to teach is to learn and to learn is to teach

meow04:09:30

it was a test to make sure I really understand what I'd like to think I finally understand well enough

meow04:09:06

and I noticed things along the way that I didn't really understand as well as I thought I did, so it was good for me

meow04:09:21

also, I used a transducer today that was a missing piece that I was pulling my hair out over until I found it and, oddly enough, it is brand new to clojure 1.7 and it's just a tranducer, not a modified func that returns a transducer

meow04:09:39

I'm willing to bet most folks don't know about it.

teamaverage04:09:42

I'm reading through this, currently up to the "Transformers" section http://kukuruku.co/hub/funcprog/clojure-transducers-reducers-and-other-stuff

meow04:09:35

yeah, that's one of the better articles

meow04:09:33

dedupe is also new to 1.7 and is a nice, simple example of a stateful transducer

meow04:09:38

random-sample is pretty cool and should be fun to use

teamaverage04:09:39

Hrm okay, so how do you compose transducers?

teamaverage04:09:58

(map +) (filter odd?) ??

teamaverage04:09:05

Oh, that easy

meow04:09:33

what's kind of cool is that transducers aren't really a thing - they're just functions that play by the rules

teamaverage04:09:40

I think I'm gettin' it

teamaverage04:09:51

Yeah, i like they're not some new compiler trickery

meow04:09:52

same old functions

teamaverage04:09:57

Just a new way of thinking

teamaverage04:09:15

I get blown away by the "meta-ness" of it all ha ha

meow04:09:57

but once you get it you realize its a pretty small amount of ground that's covered - there isn't that much to learn

teamaverage04:09:11

I think actually my problem is/was how to get transducers composing

meow04:09:35

and the source code of clojure core makes it so much easier to comprehend

teamaverage04:09:37

Cause for ages you're talking about reducers of (acc input -> result)

teamaverage04:09:04

but transducers are ( x -> x') or whatever

teamaverage04:09:44

Cause i thought there must be some unwrapping of transducers to get the reducer out ..

teamaverage04:09:53

I think I just need a break and to let it settle in

meow05:09:06

I've only been working with clojure for a little over 3 months so I don't really have a history with reducers - I just jumped right into transducers

teamaverage05:09:12

And you're most likely pulling your hair out ha ha

meow05:09:55

not at all - good review for me

meow05:09:12

but it is no longer today, here, so time for some sleep

teamaverage05:09:11

It's pretty nice to have all your functions in the one place; not spread between parens

danielcompton05:09:29

What’s the name of the Clojure site that shows usages of a function, libraries dependencies, who uses who, e.t.c.? I always forget the name

danielcompton05:09:13

nope, it’s more interactive than that, it shows source code too

sofra05:09:52

no problem

seantempesta07:09:06

has anyone used rethinkdb for streaming changes to web clients? I’m exploring using it in a project and am running into conceptual problems.

meow11:09:40

@seantempesta: you might ask on the #C073DKH9P and/or #C0620C0C8 channels as there seem to be several folks there that are using rethinkdb

tel12:09:15

@meow: the early stopping thing is pretty simple, too

tel12:09:27

there’s just a type called Reduced

tel12:09:33

which wraps the reducts

tel12:09:51

you create the wrapper by calling (reduced r)

tel12:09:11

and when transduce is performing the final reduce

tel12:09:15

it’s a little special

tel12:09:59

because it will stop as soon as it sees the reduct is wrapped in Reduced

tel12:09:08

so you can terminate the reduction early

tel12:09:47

pkobrien: also, cat is “just" (mapcat identity) simple_smile

meow13:09:03

@tel: What I don't understand is when I would need to terminate the reduction early. Got a canonical clarifying example of when one would need that?

meow13:09:34

cat is "just" (mapcat identity) - hmm, interesting

meow13:09:18

yes, I suppose that's the way to look at it

meow13:09:42

although it is shorter and when I found it something just clicked and so I used it

meow13:09:12

gotta love these things - they infect your brain - in good way

tel13:09:25

if you didn’t have the ability to force early termination

tel13:09:52

then take’s reducer would just be (fn [a r] r) every time after you’ve passed enough elements

tel13:09:08

which means you still have to iterate over the whole source container

tel13:09:23

but with early term you know when to stop because you’re never going to affect the result again

tel13:09:44

and yeah, I’m not saying to replace cat with mapcat identity

tel13:09:00

just to note that it arises really naturally from the core operation of transducers all together

meow13:09:15

right, duh! take is a great example

meow13:09:20

and I got what you meant about cat / mapcat identity relationship

meow13:09:37

@tel: very helpful, thanks simple_smile

anthonyrrubin21:09:41

"Haha, no. I had high hopes for Clojure for a while, but they're fairly user-hostile, even if they think (and loudly assert) that they aren't."

meow21:09:26

@anthonyrrubin: not sure what would make him say clojure is user-hostile - I've only been using it for 3 months and haven't found that to be the case

anthonyrrubin21:09:42

I do not know.

meow21:09:28

I'm not going to lose any sleep over it, that's for sure. Plenty of very helpful people in this community.

ecelis21:09:32

Users always think developers are hostile towards them 😛

puzzler21:09:41

Yegge's comment dates back to his interaction with the Clojure community years ago. He had a lot of ideas for improving Clojure, especially with respect to making it more amenable to tooling, all of which were ignored. The discussion basically came down to... Yegge: Clojure is not sufficiently receptive to new ideas. ClojureCommunity: If we added everything that people suggested, the language would suck. Both sides are true.

jsyeo22:09:27

@anthonyrrubin: he had a lot of nice things to say about clojure in his foreword to Joy of Clojure