Fork me on GitHub
#core-async
<
2020-05-29
>
rickmoynihan11:05:08

So I watched this talk by Michael Sperber the other day on Concurrent ML: https://youtu.be/pf4VbP5q3P0?t=1573 Where he essentially argues that CML is one of the best/forgotten abstractions for concurrent programming (summarising). Towards the end of the talk he tries to draw some comparisons to other approaches, one of which is core.async; where he makes some criticisms of it; that I don’t quite understand. Can anyone offer some wisdom on what the benefit of CML over core.async is, and what exactly is Sperber saying that core.async got wrong?? My interpretation (summary) of what he’s saying seems to be is:

rickmoynihan11:05:05

1. Having combinators on read actions but not on any of other actions… What would the other actions be? Presumably putting stuff on channels, what else? What would such combinators look like? Also isn’t alts! a combinator on putting too?!

rickmoynihan11:05:12

2. core async complects lightweight concurrency (threads/fibres) with message based concurrency

rickmoynihan11:05:20

3. A vague assertion that core.async looks like it might work but in practice it kind of doesn’t

rickmoynihan11:05:24

4. It gets backpressure wrong in a way I don’t quite follow… I think he’s saying that there are 3 buffers (a queue of threads trying to send, a buffer on the channel, and a queue on threads trying to receive). What is he arguing here, that there should just be one buffer on every rendezvous not three?!

rickmoynihan11:05:25

Sperber seems to be trying to say something here, but I don’t really follow it… Does anyone have any experience with CML, and why it might on some axis at least be superior?

mpenet11:05:17

4. he's talking about pending take queue and pending put queue I guess, but there are ways to affect back-pressure he's not mentioning like the ret val of various flavors of put/takes and derivatives (offer, poll etc). You have way to know when flow control has to happen and act on it yourself in some cases, arguably it's not simple but then again it's better to have the knobs than not

mpenet11:05:20

core.async is quite low level, maybe that's the complain. There are many ways to shot one-self in the foot and the docs (or lack, thereof) do not help. But overall I do not agree personally with his point. Not sure about the superiority of one approach vs the other, I think it's also quite situational

mpenet11:05:06

like actors vs csp, depends on caracteristics of the platform (vm), problem to solve and so on. No silver bullets

pmooser11:05:15

I feel like core.async often feels like half of a solution, and it doesn't feel like it's really under very active development. I use it all the time, but it often feels like you have to be quite careful, and there is a lack (in my opinion) of non-trivial examples of its use, especially things showing a realistically robust lifecycle.

mpenet11:05:02

I quite like it in current shape, modulo the few bugs. I don't know of a super easy to use solution to these kind of problems personally, erlang is full of traps too for instance and I suspect the same is true for cml and others

pmooser11:05:53

I like it quite a lot as well, to be honest. I think if we were able to make progress on some long-standing bugs, and if we just cleaned up or closed old tickets that were never going to be addressed, it would help the impression a bit. Like, tickets like this exist, even though the last time I discussed it with someone they thought it might not even be possible: https://clojure.atlassian.net/projects/ASYNC/issues/ASYNC-27?filter=allissues&amp;orderby=priority%20DESC&amp;keyword=line%20numbers

pmooser11:05:07

But from reading that ticket, you might have the impression that it should already be working in CLJ!

pmooser11:05:05

I mean, I'm still shocked that logical operators don't short-circuit in go blocks, and it is months since I tripped over that one.

rickmoynihan12:05:07

WAT?! Really?!

rickmoynihan12:05:02

(clojure.core.async/<!! (clojure.core.async/go (or nil (println "hello") 10)))
hello
10

(clojure.core.async/<!! (clojure.core.async/go (or 20 (println "hello") 10)))
20

rickmoynihan12:05:29

looks like they do

Alex Miller (Clojure team)13:05:29

it's a bug in cljs only iirc

rickmoynihan13:05:28

ah… not the end of the world

pmooser15:05:28

Yes, my apologies for not being specific enough. Most of the libraries I write at my company where we use core.async end up being used in both clj and cljs, so I tend to muddy these two together. But I don't think the bug is any less severe for only affecting one target platform. You could argue that it's worse because it's a significant behavioral change between the two.

rickmoynihan15:05:48

Sure — It’s probably a painful one to debug; but relying on short circuiting to prevent effects isn’t that common, and once you know you can work around easily enough at least.

mpenet11:05:52

it needs love for sure.

mpenet11:05:09

but I'd rather have spec2 first, so I won't complain too much about it 🙂

mpenet11:05:16

given the current bus factor

pmooser11:05:29

I mean, it's a volunteer project, so it isn't like we can really complain about it anyway. I'm just talking about wishes.

pmooser11:05:58

I'll be interested to see where spec gets to eventually, but last time I checked it was very functiony and not very data-y.

pmooser11:05:13

I should read up on spec2.

mpenet12:05:26

it's debatable. spec2 has a "data" layer, then "forms" might be off-putting but technically it's data 🙂, ultimately there's a predicate at the leaf no matter the approach

mpenet12:05:04

but who knows what it's going to look like in the end. Apparently it's still in flux, we ll see

pmooser12:05:04

Yeah. I have a quite a bit of faith in their design process. Just need more patience I suppose.

mpenet12:05:32

a good read about flow control (it's erlang flavored but a good read) https://ferd.ca/handling-overload.html

pmooser12:05:13

Thanks - I will give it a look.

rickmoynihan12:05:12

Yeah, I don’t have any particular complaints about core.async, but I only use it in a few places where it’s been great. I did get into Erlang about 15 years ago; but eventually grew to dislike it and the actor model. I definitely like that clojure has multiple idiomatic approaches to concurrency as I’ve previously also felt there’s no one true way to do it. However the CMLers seem to really believe it gets a lot right that other models don’t, and I’m curious what that is and what it might look like… It sounds like it might be a bit like the blub paradox… I suppose I’ll need to buy the book to find out…

hiredman16:05:04

my impression of the main difference between the CML model(I've never actually used CML, but have read papers an played with implementing the api in clojure) and core.async's more CSP style api is that CML's notion of events make first class and explicit things that are kind implicit in core.async. And making events first class allows for all kinds of interesting usage (events as capabilities, composition of events, etc), and makes it possible to extend and add behaviors to existing events using guards, nacks, and wrap

hiredman16:05:38

guards and nacks are particularly interesting for integrating with external services and allow you do things like create some kind of event that represents reading a message from rabbitmq, and if someone synchronizes on that event it goes out to fetch a messages, but if the synchronization was part of an alt (select) and that event isn't chosen, automatically nack the fetched message

rickmoynihan16:05:21

Thanks, this seems to tally with what he was saying earlier in the talk. I did find a CML implementation on top of core.async… I’ll need to take a look at it: https://github.com/polytypic/poc.cml

hiredman16:05:20

first class events make the CML api much more powerful because it sort adds a loop to the graph of opertions

4
hiredman16:05:59

in core.async with a channel, you can >!, <!, or alt, you get some value out

hiredman16:05:45

in CML when you take or put you get an event, and select takes events and returns an event

rickmoynihan16:05:06

Interesting…

rickmoynihan16:05:34

So the other thing he says is core.async complects fibres with message based concurrency… I mean sure, I can see that… But to implement lightweight threads; don’t you have to complect them with a communication mechanism?! Presumably CML complects in a similar way too; is it just that its event based mechanism is considered superior?

hiredman16:05:36

I dunno, I haven't watched the video, but yes cml is all kinds of complected, it is a whole other dialect of ml with a language runtime written to support the cml api

rickmoynihan16:05:46

Sure, though it actually seems most CML implementations aren’t actually in ML… e.g. guile has one, racket, *haskell etc

hiredman16:05:04

those languages/runtimes all have green threads independent of their cml implementation I believe

rickmoynihan16:05:47

totally, but if you’re implementing green threads (like core.async had to) don’t you also need to implement a way to communicate with them? Put another way what might core.async style green threads look like if decomplected from channels?!

rickmoynihan16:05:14

thanks for connecting those dots for me

rickmoynihan16:05:15

Are there any other implementations of co-routines in clojure that you’re aware of?

hiredman16:05:06

the go macro in core.async is more or less writeing the code to make delimited continuations work, the go macro itself establishes a reset, and each channel op is a yield

hiredman16:05:44

I am sure there are others, I've got one not published and half done

hiredman16:05:20

and about 10% of an http2 server written that uses a continuation monad for similar effect

hiredman17:05:23

a continuation monad is kind of a neat way to get continuations without needing a complicated macro like core.async, but it does make your code look really weird, and if anything makes the stacktraces even worse

rickmoynihan17:05:06

Yeah roll on project loom 🙂

rickmoynihan17:05:14

continuation monads are a neat trick, that’s for sure — thanks for the other pointers too 🙇

noisesmith14:06:57

this thread is very enlightening to read - it makes so much sense, ml and scheme are "coroutine complete" (call/cc being a building block for proper coroutines) so it makes sense they can do green threads in a less complected way

👍 4
hiredman16:06:02

in fact the work for those languages tends to be the other direction, adding support for real threads http://manticore.cs.uchicago.edu/papers/icfp09-parallel-cml.pdf is a paper from project manticore over ten years ago (which is sort of the spiritual home of cml I guess?) about a cml implementation with real threads

rickmoynihan09:06:40

Yeah manticore seems to have been Reppy’s work at applying his earlier CML work to real threads for parallelism, rather than concurrency.