Fork me on GitHub
#core-async
<
2015-07-23
>
magnars16:07:29

the core.async docs lists a bunch of deprecated functions, like map< and filter<, saying "Use transducer instead". But there doesn't seem to be any function that takes a channel and applies a transducer to it. What have I missed?

alejandro16:07:08

@magnars: (chan buf xf) will apply the xf to everything that passes through the channel

magnars16:07:44

yes, but that conflates creating the channel and applying a transducer to it - which fails in my case, since I'm not the one creating the channel

alejandro16:07:50

fair enough. In that case you may need to splice it in yourself

alejandro16:07:23

can use pipe with a channel of your own creation to get the transducer application?

magnars16:07:25

yeah, that is certainly doable - sounds like something that would fit into core.async core tho

magnars16:07:08

(let [c (chan 1 (map :message))] (pipe ws-channel c) c) isn't exactly the epitome of elegance

erik_price17:07:10

useful as it can be, core.async isn’t exactly the epitome of elegance

magnars17:07:42

true that, but this conflation of creation and transformation seems to be at odds with Clojure's notion of simplicity

erik_price17:07:08

that’s actually what i’m referring to. Although there’s ways of mitigating some of the pain, core.async is really the first time I’ve felt like a core Clojure library embraced imperative APIs, mutability, and identity. It’s a great tool, but it feels distinctly different from so much of the rest of Clojure.

robert-stuttaford18:07:10

well, core.async channels are stateful. so, there’s that

ghadi18:07:41

The point of channels is conveyance... processes are inherently harder

ghadi18:07:12

I see it half-glass-full in that channels help simplify subsystems

robert-stuttaford18:07:39

certainly better than writing callbacks

magnars19:07:41

my intention was not to point out flaws inherent in core.async, just a wish for a function that applies a transducer to an existing channel simple_smile

ghadi19:07:13

there's no way to do that without a race condition

magnars19:07:45

sorry, let me rephrase - returns a new channel with the transducer applied

magnars19:07:13

I basically want to do (map< my-fn my-chan) with transducers, something like (<xf (map my-fn) my-chan) maybe.

ghadi19:07:46

Yeah, I think make a new channel and pipe is your best bet

magnars19:07:17

yeah, one problem with (let [c (chan 1 (map my-fn))] (pipe my-chan c) c) is that it'll take five minutes out of my video to explain it, while the deprecated map< would not. It also seems to me like a good fit for a basic feature in core.async.

magnars19:07:58

it seems like such a basic thing to do, that I'm surprised it's not there

ghadi19:07:08

¯\(ツ)

ghadi19:07:27

libraries usually prefer finding primitives...

ghadi19:07:34

rather than conveniences

magnars19:07:52

wouldn't "decomplecting" creation and transformation be simpler to work with, in a non-convenient sense?

ghadi19:07:39

I don't follow

magnars20:07:04

let me try again simple_smile creating a channel is a primitive operation. In my mind, applying a transducer to a channel is a primitive operation too. Please note, this would of course not change the original channel, but return a new one, just like the rest of clojure works. It's obvious the core.async-creators thought along the same lines when they created map< and filter< etc. Then transducers came along, with a deprecation "please use transducers" - but those were bolted on the chan constructor, complecting together channel creation and transformation.

magnars20:07:36

so if you want to, say filter a channel, you have to create it yourself.

magnars20:07:52

you dismissed this by saying that what I was asking for was just a convenience (easy, in Hickey-parlance), but my point is that it would be conceptually simpler.

erik_price20:07:23

Our codebase uses core.async in a lot of places, and we’ve written a lot of higher-level functions similar in spirit to the one you would have to write to get this behavior.

ghadi20:07:30

magnars: what happens when you apply a transducer to a channel with a transducer already?

ghadi20:07:27

magnars: not saying we'll never change the API, but especially with async things there's a lot of subtleties

magnars20:07:42

I'm not saying "apply the transducer to the channel itself"

ghadi20:07:47

ah, I must be missing something.

ghadi20:07:45

can you post some pseudo-code?

magnars20:07:42

I guess what I'm asking is, could something like this be part of the core.async API:

(defn <xf [ch xf]
  (let [c (chan 1 xf)]
    (pipe ch c)
    c))

magnars20:07:59

possibly implemented in a less naive fashion

magnars20:07:37

as it stand, map< has been deprecated with no real replacement

ghadi20:07:06

it probably won't be replaced

ghadi20:07:28

merely because you can do it just like the code you posted

ghadi20:07:56

(`map<` had other bad issues too regarding exceptions)

magnars20:07:57

so you don't think "applying a transducer to a channel" is a primitive operation?

magnars20:07:30

I'm gonna need some time to wrap my head around that, but fair enough

ghadi20:07:59

it's different than seqs in core because channels aren't datastructures

ghadi20:07:09

there's no notion of persistence

magnars20:07:42

but channels are first class citizens, in that they can be passed around - and now transducers are only added at the point of creation

magnars20:07:08

I can't get a "firehose" channel, and then filter it based on some criteria

magnars20:07:45

without going through hoops, that feel to me highly stateful, with pipe

ghadi20:07:09

but channels have state

rauh20:07:09

Btw, pipe returns you to channel so you don't need the let (pipe my-ch (chan 1 (map inc)))

magnars20:07:17

maybe I'm thinking about channels too much like "Rx"

ghadi20:07:30

rauh: pipe's return signature may change in the future to return a channel that closes when the piping process is done

ghadi20:07:03

pkobrien: that is also subsumed by transducer and will disappear

meow20:07:15

or is filter also deprecated...

magnars20:07:18

@meow: yeah, that's the one I want, except it would take the transducer as an argument, but it's deprecated and not getting a replacement

meow20:07:29

@ghadi: ok, yeah

magnars20:07:58

but I've only been working with core.async a few months, and am possibly thinking badly about it

magnars20:07:27

I'm thinking about it as a stream of events, and I'd like to filter/map those events

magnars20:07:06

but doing those transformations does not feel supported in the core.async API today

meow20:07:18

well, its easy to put a transducer on the channel that you create, but you know that

magnars20:07:30

except I'm not the one creating the channel

magnars20:07:35

or some other part of the code could be doing that

magnars20:07:46

using pipe the way @rauh mentioned looks OK, tho, but the return value of pipe is apparently not part of the official API atm

rauh20:07:58

Yeah but you could easily re-do pipe that still behaves that way in case it does change.

magnars20:07:12

sure, I could also easily add the <xf function I pasted earlier simple_smile but since this seems "obvious" to me, but not the people working on core.async, I'm left with the feeling that there is some central idea that I've not grasped yet. Which is what got me here in the first place.

magnars20:07:42

and after this, I'm left with a vague notion - something like this: since channels are stateful, they don't compose as well as data structures

magnars20:07:12

also, using a channel someone else (some entirely other part of the code) created isn't done very often

meow21:07:05

@magnars: so you've got one part of the program creating a channel that is to be passed to another part of your program for the other part to do puts onto the channel, yes?

meow21:07:16

I'm still learning the best way to use core.async myself, so just thinking aloud in case it helps.

magnars21:07:22

in my case, I'd just like to (map< :message my-ch) before passing the channel to the consumer of the channel, to remove knowledge of the entire datastructure to the consumer.

magnars21:07:25

then I noticed map< was deprecated, went to look for the replacement, found only chan, which I could not use, since I didn't create my-ch in the first place

magnars21:07:17

I then started using pipe - but was still curious why my intuitions about channels were so wrong

magnars21:07:55

ie, why didn't the maintainers of core.async think creating a channel and filtering a channel were two separate concerns, simpler apart than together

meow21:07:34

One thing I'm curious about is what is the main benefit of using a transducer as opposed to simply passing a value to some function before putting it on a channel since you can't have transducers on the way out, and in the wild it seems like a lot of the examples I've seen have been just that - function calls on the way in or out, rather than transducers.

magnars21:07:48

it's a composability-issue - maybe I have a channel with lots of events on it. Before I pass it to you, I want to filter in only the events relevant to, say, a single user.

magnars21:07:43

that might be the domain of the auth-code - and should be kept there

rauh21:07:43

I'm not sure I understand you completely. What do you mean with "found only chan, which I could not use, since I didn't create my-ch in the first place"?

rauh21:07:26

Think of channels simply as FIFO buffers that you interact with, you can't have different "views" on them. You'll need a new channel

magnars21:07:51

chan was the only place to "add" a transducer

meow21:07:55

meaning you can specify a transducer when you create a channel

rauh21:07:35

The thing with anything that modifies a channel is: You need a new channel for this anyways. Look at the map< implementation. It also created a new channel.

magnars21:07:56

yeah, like I'm saying: I don't want to modify a channel in-place, that would be even more un-Clojure-like

meow21:07:01

but I see that transducer when you create the channel as just sort of a convenience, since like rauh says they are just FIFO buffers

magnars21:07:21

I'd just like a way to say "hey, map this channel with this function" in a way that feels natural and supported

rauh21:07:47

But then what do you want the return value to be?

magnars21:07:51

Imagine if you had to map over a vector in clojure.core with something like (let [c (chan 1 (map my-fn))] (pipe my-chan c) c)

magnars21:07:01

the new channel

magnars21:07:41

I basically want this in the API:

(defn <xf [ch xf]
  (let [c (chan 1 xf)]
    (pipe ch c)
    c))

rauh21:07:43

Oh I think I got you

rauh21:07:46

I guess you'd have to create that convenience API yourself

magnars21:07:40

indeed - and that's fine. But it seemed like such a basic thing to be doing, that I was surprised it was not in the API.

magnars21:07:00

which led me on this quest in this channel to understand why it is in fact NOT a basic thing to be doing.

magnars21:07:27

because I'm making a video, and I'd like to use core.async in an idiomatic way

rauh21:07:05

Well one thing is that you gain very little by this convenience function and then you'll have the argument "if this function made it into core.async then I also want function xyz"

magnars21:07:44

again, in my mind, the functions map< filter< distinct< etc were remove since they were "redundant", and transducers came to the rescue

magnars21:07:15

but there is no plan to actually give us the transducer-version of these functions

rauh21:07:18

Which is often an argument that some convenience function don't make it into core libraries, to prevent getting tons of other functions that do very little

rauh21:07:00

E.g. the chan version adds buffers which are more powerful. You can do the usual >1 buffer size or dropping buffers etc

magnars21:07:47

I realize it is my own limited understanding here, but it sounds to me like you're saying "map and filter are only convenience functions, because you can do both of those with reduce, and we don't want to go on a slippery slope"

magnars21:07:45

and that is true, in a way, but you'd always have "map" and "filter" anyway, because they are core - and in my understanding, so would "give me a new channel, with this channel as input, transduced like this" be - core.

magnars21:07:13

but I don't want to waste more of your time on this. Thanks for your patience. I'll show myself out. 😛

rauh21:07:22

I guess it could be cool to add another arity to chan and allow a from-chan to make things nicer, but chan already has so many arguments ;/

rauh21:07:18

Or (chan from-chan (map inc)) would be cool and consider a from-chan to be a "form of buffer" simple_smile

rauh21:07:02

@magnars: But I doubt any of this is going to happen.

magnars21:07:02

yeah, I'm not hoping for change - I'm hoping for insight. At this point, I've got two points of insight out of this: 1) map, reduce, filter might be basic operations on event streams, but not for channels. The core.async team does not think about channels as event streams, even if I do. I think here is the mismatch. 2) my ability to articulate an idea in textual form is sorely lacking.

erik_price22:07:14

what about (clojure.core.async/map :message [my-ch])?

ghadi23:07:50

magnar: speaking for myself, I think channels are more generic than event streams (encompassing alt! and other coordination)

ghadi23:07:02

you can do event-streamy things, but a lot more