Fork me on GitHub
#core-async
<
2016-02-12
>
bbloom02:02:24

@alexmiller: Please don’t put closed? in the core.async API simple_smile

bbloom02:02:41

I’ve been writing a lot of Go lately. The biggest problem with core.async is that you can’t create sane abstractions without exposing channels. Double bad b/c you have to expose read/write channels.

bbloom02:02:56

I would really like to see dynamic enforcement of read vs write ports

bbloom02:02:10

that would prevent (via an exception) writing to read-only ports and reading from write-only ports, plus closing read-only ports

bbloom02:02:53

After all this Go code I’m writing, if i were to return to core.async, I’d probably be writing a fair number of macros for the public interface of my abstractions

bbloom02:02:09

since you want functions that return values w/o the channel in between

bbloom02:02:18

exposing a channel makes it super easy to screw up a concurrency invariant

bbloom02:02:24

For example, the Close method on the Subscription type in the examples in ehre:

bbloom02:02:42

Close returns error, not a chan of error

bbloom02:02:04

you can’t make functions that do async work b/c of the lexical go macro

bbloom02:02:45

so i’d probably implement something like (defmacro unsubscribe [subscription] …)

bbloom02:02:03

which is unfortunate

bbloom02:02:15

but not a big deal in imperative code anyway

alexisgallagher04:02:19

@bbloom Are you saying it's bad to put channels at interface abstractions in general? For instance, do you think that Subscription example would be better if it did not expose a channel via Updates ?

bbloom04:02:36

@alexisgallagher: no. note the type of the Updates method’s return value

bbloom04:02:40

it’s a read-only channel

alexisgallagher04:02:17

Okay. I was reading your "The biggest problem with core.async is that you can’t create sane abstractions without exposing channels. Double bad b/c you have to expose read/write channels." and wondering if it was still "single bad" in your view to expose channels at all.

bbloom04:02:33

whoever is publishing to that channel can do:

bbloom04:02:35

select { case sub.updates <- update: // successfully published default: // handle slow subscriber }

bbloom04:02:00

or alt!/:default in clojure

bbloom04:02:31

it’s single bad to expose a channel if you don’t document how to use it simple_smile

alexisgallagher04:02:39

Ok, that makes sense.

alexisgallagher04:02:49

I've never worked with channels before, but I dove in head first last Monday on a piece of work, so I'm still finding my bearings.

bbloom04:02:54

holistically, i don’t think you can enforce everything either statically or dynamically

bbloom04:02:24

but concurrency is hard, so i’ll take all the help i can get

alexisgallagher04:02:33

I agree with that one.

bbloom04:02:36

hence my desire for enforcement of read/write-only ports

bbloom04:02:50

note that i’m happy for that enforcement to be dynamic (exception), rather static (type error, a la Go)

alexisgallagher04:02:00

Yeah, makes sense to me. But I've been marinated in Swift for the last year or so so I've become much fonder of type systems than I ever expected to.

alexisgallagher04:02:54

I'm still trying to get a sense of where channels add something distinctively superior to other options, and where not. I'm not sure if they should be treated as something you only fall back to for complex cases, or something that should be used universally.

bbloom04:02:24

depends dramatically on your application

alexisgallagher04:02:27

For instance, a few hours ago, I realize that I'd way overcomplicated what I've been working on lately, by adding an unnecessary channel in the middle, but that's probably just because I'm new to them.

bbloom04:02:36

i’ve found quite a few uses for them when dealing with lower level stuff from Go

alexisgallagher04:02:05

lower level like byte streams?

bbloom04:02:11

I’ve used a ton of channels and go concurrency patterns for a web service that interacts with FFmpeg and some other services, and now again for a Go application that does a lot of bluetooth low energy stuff

bbloom04:02:25

mostly, i’m using channels to make bad C interfaces less stupid to deal with

bbloom04:02:54

i’ve advised strongly lots of people to remove core.async from large parts of their CLJS apps

alexisgallagher04:02:08

Interesting. I remember hearing there was evn a CSP implemetnation in C (libmill) but never figured out how solid or complete it was.

bbloom04:02:26

no idea, i’m using cgo

alexisgallagher04:02:19

What I spent a hunk of today realizing is that close! should not be used as a form of back pressure, or for consumer to producer signalling. But this just opened the question of how to tell a producer to shut up. Another channel? Or just set a flag?

bbloom04:02:50

another channel

alexisgallagher04:02:58

My stumbling journey to enlightenment unfolded in #C03S1KBA2. Didn't know about #C05423W6H.

alexisgallagher04:02:29

Why another channel? Just for aesthetic consistency? Or is a (chan (dropping-buffer 1)) just to receive a :stop command functionally better than an atom with a stop flag?

bbloom04:02:42

most concurrent processes look like that

bbloom04:02:49

one big select where every type of event gets its own channel

alexisgallagher04:02:18

Right, that's one way I was wondering about, (alts! control-channel other-chan :priority true)

bbloom04:02:19

separate channels mean separate orders

bbloom04:02:28

no need for priority in most cases

bbloom04:02:51

note that example has two reads and one write

alexisgallagher04:02:04

When i was studying timeouts it seemed to me you'd always want priority or else you face the improbable but possible scenario of your timeout never getting tried and found closed.

bbloom04:02:25

the point of select / alt! is to deal with blocking

bbloom04:02:01

concurrent blocking, specifically

bbloom04:02:11

Go doesn’t offer a priority open

bbloom04:02:16

you have to make nested selects if you want priority

bbloom04:02:21

i’ve only ever really needed that once, maybe twice

bbloom04:02:35

and i’ve written dozens of complex async machines at this point

bbloom04:02:57

personally, i think core.async’s priority flag is a misfeature

bbloom04:02:06

b/c nesting already exists & is much clearer

alexisgallagher04:02:10

even though it makes it deterministic ?

bbloom04:02:22

CSP is about non-determinsism

bbloom04:02:43

if you have determinism and want to preserve it, do not use CSP

alexisgallagher04:02:53

Right, but shouldn't the choice of timing out if possible but guaranteed once it is possible ?

bbloom04:02:10

i don’t understand the question/problem

bbloom04:02:37

if you alt! on some channel read and a timeout channel read, and the first channel blocks forever, the timeout read will proceed

alexisgallagher04:02:28

My worry was if you had (alts! (timeout 100) chanA chanB ... chan1000) where all channels were available, then it becomes less probable alts will "look" at timeout, and then the effect of the timeout is delayed.

bbloom04:02:47

um, just don’t create a new timeout every time you call alts!

bbloom04:02:50

just store your itmeout channel somewhere

bbloom04:02:05

if you want a global timeout

bbloom04:02:25

if all your channels are always ready, then they aren’t timeing out

bbloom04:02:36

you have to differentiate between a read timeout and a process timeout

alexisgallagher04:02:49

Ah. Ok. New distinction for me to mull over.

bbloom04:02:11

you can use the same timeout channel for 3 or 4 different read operations

bbloom04:02:14

if you do like a waterfall

bbloom04:02:19

ie read from A then from B and then from C

alexisgallagher04:02:24

That makes sense. So if I'd only put the timeout in the same alts as the read channels if I wanted to force a read timeout. If I want a process tiemout then I put it one level higher.

bbloom04:02:32

you can have some timeout channel T and you can do alt:A/T, alt B/T, alt C/T

bbloom04:02:37

and each alt can timeout

alexisgallagher04:02:52

Thanks, this make sense. And yeah from that POV I can see why :priority seems a bit superfluous. Since you have an ordering mechanism when you really want it, and when you don't want it you really shouldn't be expecting it at all.

alexisgallagher04:02:46

No surprise, but seems like the go community has a lot of deep working knowledge on how to work in this style.

alexisgallagher04:02:52

That talks looks quit good.

bbloom04:02:10

most interesting concurrent go programs are a bunch of little machines, each one with a bunch of heterogenous channels and a go-routine running a for loop around a big select (alt)

bbloom04:02:23

plus a public API that shields the consumer from those channels

alexisgallagher04:02:36

Interesting. When i was thinking of "control channel to shut off polling" I felt like I should wrap it in api by having a function like (stop-polling [w]). Do these APIs often the flavor of start/stop/adjust commands embedded in function names?

bbloom04:02:21

the subscriber pattern in those slides is pretty common

bbloom04:02:36

it’s a tad more awkward in clojure b/c you cna’t hide channel reads and writes

bbloom04:02:55

b/c the various read/write/alt/etc operators are magic macro special form thinggies

bbloom04:02:06

as opposed to real threads

bbloom04:02:30

on the JVM you can use >!!, <!!, alt!!, but only on real threads

bbloom04:02:39

which kinda defeats some of the purpose, but is ok for some use cases

bbloom04:02:59

otherwise, you basically just need to return a map of channels and write good doc strings on what you can write to which channels when

bbloom05:02:41

so instead of something like .Subscribe() returning a Subscription object with an Updates() method, you’d instead return something like {:updates #<a channel> :stop #<a channel>}

bbloom05:02:48

and then document your subscribe method as such: “returns a map with two keys. The :updates value is a channel yielding all the changes, until you close the :stop channel”

bbloom05:02:21

and then you rely on the “we’re all consenting adults here” policy of dynamic typing to not do anything else to either channel (eg close the updates channel or write to the stop channel)

bbloom05:02:30

or read from the stop channel, heh

alexisgallagher05:02:11

I like the idea of a stop channel that's not used to send a special value :stop, but where the client just sends the only message needed by close!ing the channel. That's slick. No faff around deciding the magic token.

bbloom05:02:45

if you need something more complex than stop, you can use a control channel and just put keywords on it

bbloom05:02:54

in Go, it’s more common to use multiple channels b/c it’s annoying to make constants

bbloom05:02:01

we have keywords in clj, so just use them if you need em

alexisgallagher05:02:09

So why would it be better for (Subscription) to create and return the channel itself, rather than take the channel as an argument and just build the go block that populates it?

bbloom05:02:15

{:updates #<a channel> :control #<a channel>}

bbloom05:02:15

"returns a map with two keys. The :updates value is a channel yielding all the changes, until you close the :control channel. You can also send [:subscribe 123] or [:unsubscribe 456] to add or remove new records to the scope of your subscription”

bbloom05:02:22

or something like that

bbloom05:02:18

it’s not better or worse to accept a channel param vs create your own param

bbloom05:02:21

depends on use case

bbloom05:02:40

often it’s nice to take channels as input b/c it means callers don’t need to do as much alt! etc

bbloom05:02:01

b/c they can let publishers do the merging on to one channel implicitly by nature of accepting that channel as a parameter & using it

bbloom05:02:28

but now you have to worry about things like buffer sizes with concurrent writers

bbloom05:02:32

always tradeoffs

alexisgallagher05:02:18

Rather than defining a mini language of special keyword values for the channel, would it be typical/advisable in public API to instead just defining functions (subscribe m num), (unsubscribe m num), etc., that wrapped the dirty details of sending value X onto the channel under keyword Y. ?

bbloom05:02:30

you can’t do that

bbloom05:02:35

not in clojure anyway

bbloom05:02:45

a go block needs to be able to “see” a read/write/etc

bbloom05:02:52

it has to be lexical

bbloom05:02:59

hence my comment earlier about macros to do such a thing

alexisgallagher05:02:18

I confess: I didn't understand that comment. 😳

bbloom05:02:22

the most clojur-y thing to do would be to just do what im saying with messages like [:foo 1] or [:bar 2]

bbloom05:02:04

i was expressing my wish to alexmiller that i could get slightly better enforcement for read vs write ports

alexisgallagher05:02:04

Hmm, but why not (defn subscribe-num [m num] (go (>! (:updates m) num)))

bbloom05:02:12

b/c now that’s an async send

bbloom05:02:25

(go (>! or (go (<! are always wrong

bbloom05:02:33

(go (>! is put! and (go (<! is take!

bbloom05:02:35

both are async

bbloom05:02:38

not blocking

bbloom05:02:44

no backpressure

bbloom05:02:57

you will ultimately run in to the go routine limit

alexisgallagher05:02:20

I didn't mean :updates. I meant the control channel..

bbloom05:02:42

sure, you can do async sends to the control channel and it’ll probably be fine for small number of control messages

bbloom05:02:57

but it will make it harder to reason about your concurrency properties

alexisgallagher05:02:09

You mean within the loop in the go block ?

bbloom05:02:28

if you have an unbuffered “stop” channel in your go loop, then a successful write to that channel means that the stop message was delivered

bbloom05:02:29

for example

alexisgallagher05:02:19

I was earlier imagining a (chan (dropping-buffer 1)) control channel that just sat around waiting for :stop to show up before you raised the idea of a "close!-only" channel. But now I'm thinking abot the case where there's a more complex control language, and wondering if then it's better to document all the valid keywords or wrap it in functions that know the keywords.

alexisgallagher05:02:34

Hadn't really thought through what kind of a buffer a more complex control channel would best have.

bbloom05:02:05

for private/internal use, just put the messages you want on the channel and write comments

bbloom05:02:17

if you have some kind of public/stable API, you can make macros that write to the channels

bbloom05:02:27

which is what i was trying to say to alexmiller earlier

alexisgallagher05:02:51

Thanks for taking the time to explain these points. Helps me put it all in perspective.

bbloom05:02:09

(defmacro stop [subscription] `(>! (:control ~subscription) :stop))

alexisgallagher05:02:11

Why macros over plain old functions?

bbloom05:02:27

b/c macros will let go see the reads/writes/etc

alexisgallagher05:02:44

AH! So the user can invoke the macro inside go.

alexisgallagher05:02:11

That is my light finally dawning emoji.

bbloom05:02:30

happy to help

bbloom05:02:38

like i said, i’ve been writing A LOT of Go lately

bbloom05:02:53

and for all the awful things about Go, and all the complaining i’ve done about it, it’s actually quite a useful tool and super nice in many ways

bbloom05:02:27

as an engineer, i love Go. as a language geek, I hate Go 😛 as a concurrency nut, I love/hate Go depending on whether the problem I have is better solved by CSP or a persistet data structures + an atom 😉

bbloom05:02:00

happy to provide further experience report details for the other citizens of #C05423W6H another time

alexisgallagher05:02:01

Sounds like you've got a real emotional roller coaster of a language there. simple_smile

bbloom05:02:04

i’m off. good luck

hugesandwich14:02:37

@bbloom very interesting thoughts, I agree with at least parts of what you said and would love to hear more. I've probably written less Go than you, but lots of concurrent code in other languages. Curious why specifically you don't advocate using core.async in cljs? (also curious about your further thoughts mentioned on Go) I use core.async quite a bit in my current app (lots of server-push and real-time stream processing and re-processing) with great success, though the majority of my "logic" client-side is essentially conveyor belts on the edges calling to other services, server, etc, then coordinating with each other. For the UI directly, I take a simpler approach similar as in re-frame with a single conveyor and a single atom for ui updates. To that end, I'd agree that using core.async to handle each and every dom event or communicate between react components is maybe not the best plan, but not sure why else you advocate against.

hugesandwich14:02:30

@bbloom I appreciate channels and a few things in Go quite a bit, but the rest of the language makes me want to kill myself including the tooling and package management at times. I suppose some of my experience is a bit rougher having not used the latest and greatest Go versions, and writing a lot of it originally in Acme (I'm probably dating myself here, also a former Plan9 user). Anyway, a lot of Go feels like the same old same old I don't like about other languages only with some better concurrency. A win in some ways, but depending on the project, not exactly the first thing I am looking for in a language. Happy thought to accept a job in Go over most other languages, so I guess that's a compliment. I'm a firm believer in the right tool for the right job so don't really care what the language if that's the case.

bbloom17:02:11

@hugesandwich: mostly i’m advocating against core.async in views in cljs. React/etc give you such nice determinism, it’s a shame to throw that away

bbloom17:02:48

core.async is fine for doing various REST/RPC etc in your networking layer

bbloom17:02:00

although that’s a way to introduce sanity for an insane API

bbloom17:02:22

the utility of core.async approaches zero when you have a sane backend API… see all the stuff dnolen has been shouting about for a while now

bbloom17:02:29

my favorite thing about Go is that it’s not Scala, which is the other major language in use on my team's backend 😛

alexisgallagher18:02:11

It's funny to think how prevalent "My favorite thing about language X is that it's not language Y" probably is. Like we're all fugitives from past language traumas.

aria_18:02:07

Anywhere I can read more in depth about CSP?

Alex Miller (Clojure team)18:02:50

Hoare's original work is very readable, imo

ghadi19:02:44

(I'm with bbloom on closed?)

Alex Miller (Clojure team)19:02:47

Strange people lurking in chat rooms handing out opinions ... that's no basis for a system of software government! Supreme executive power derives from a mandate from the masses, not from some farcical conversational ceremony.

bbloom19:02:22

I suggest we put a watery tart in charge

bbloom19:02:33

she can throw around some swords

meow19:02:35

Is this the fun channel now?

aria_20:02:40

@alexmiller: After reading a bit about CSP and the Actor model, I’ve found that CSP sounds better to me except for the fact that, from what I’ve read, it’s not fault tolerant like actors. Do you have any thoughts on fault-tolerance in CSP?

hugesandwich20:02:49

@bbloom ok, it sounds like we're in agreement on CLJS. As for Go not being Scala, totally agree. I'm a fugitive from many languages, beginning with ASM/C as my first languages along with a bit of Fortran and a dash of COBOL, then Lisp and Smalltalk, all the way through C#, F#, Java, Scala, JS, Clojure, Python, Rust, and Ruby the last few years the most among countless other (ex: Squirrel, Go, Lua, Erlang).... I can safely see almost all have left their trauma, but I remain with a great appreciation of Lisp, Clojure, and Smalltalk even though I'd reach for some others before them. Python, Ruby, and JS leave me with great trauma even though I feel comfortable in them. Let me see I have a great appreciation for the hard work of all language authors. Scala, we could go on, but I believe it was made by people who wish code was ascii art. Python traumatizing me because it was seemingly made by someone who really needs to watch Rich's hammock talk.

ghadi20:02:28

@aria_: actors aren't fault tolerant either

ghadi20:02:33

it's how you use them

ghadi20:02:51

which is a big credit to Erlang/OTP

aria_20:02:21

That’s basically where I’m at right now

hugesandwich20:02:24

I was just going to say, Erlang is giving you a lot different impl of actors and tools than Scala for example

aria_20:02:40

I like clojure more, but elixr/erlang seems better suited for web servers, which is what I’m working on.

aria_20:02:08

So I’m not exactly sure which one to pick out of the two

hugesandwich20:02:24

@aria I find I write clearer code CSP style and less "magic." That's usually because a lot of the people I've seen implement actors like to pile in all sorts of other stuff and then break things. You can make bad code CSP style too, so it's really just how you want to build your code. I enjoyed writing systems in Erlang, but I don't really like the language beyond some of the messaging related constructs and actors. I find Clojure a bit easier to be productive and more versatile. That said, you could of course write actors and CSP in almost any language, so write actors in Clojure or CSP-style with something else if you want. What I like about CSP is that it is pretty simple. Any time you are writing a distributed system or anything that needs fault tolerance, you tend to need queues anyway. Moreover, the simpler the system, the easier it is to reason about to avoid doing things that break fault tolerance. Core.async provides in-memory abstractions with channels that help you communicate and do things like exhibit backpressure. Even if you build a system in actors, you're using a lot of these things elsewhere in your code, actors or not. For true fault tolerance, you generally need persistent stores and synchronization mechanisms which are more about tools, architecture, and algorithms than they are about using actors or CSP alone. You can screw that up with anything, and most people do (see https://aphyr.com/tags/jepsen)

hugesandwich20:02:31

One of the best things I find about core.async and doing anything that needs some semblance of reliability is that for one, unbounded queues are not the default. Too many times I've worked on systems that work until they don't because it's just "add more RAM" and things like that. Redis comes to mind.

aria_20:02:05

@hugesandwich: wow great answer thanks.

hugesandwich21:02:12

@aria - no problem. You say you are working with web servers, but do you mean writing a server itself or just writing a web app?

aria_21:02:12

@hugesandwich: I mean writing an API server for a webapp

aria_21:02:50

The webapp itself is all static files thanks to frontend templating. All I really need is an API.

hugesandwich22:02:51

@aria ok in that case, either erlang or clojure is fine. If you were writing a web server itself, I'd have other suggestions for you.