Fork me on GitHub
#core-async
<
2019-06-05
>
kenny02:06:43

What happens if I create too many gos?

noisesmith02:06:52

a queue somewhere has too many callbacks in it I guess

kenny02:06:11

Will an exception get thrown?

kenny02:06:52

How many gos can one create before these unknown consequences happen?

noisesmith02:06:37

it's probably a heap issue, or maybe there's a bound on the queue

noisesmith02:06:57

someone else here probably has a real answer, for now I'm experimenting

kenny02:06:19

(dotimes [n 5000000]
  (async/go (async/>! c n)))
results in a bunch of these:
java.lang.AssertionError: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
                          (< (.size puts) impl/MAX-QUEUE-SIZE)

noisesmith02:06:47

user=> (def a (atom 0))
#'user/a
(cmd)user=> (dotimes [_ 500000] (>/go (>/timeout 10000) (swap! a inc)))
nil
(ins)user=> @a
500000

noisesmith02:06:10

adding another 0 to the end of that number made it take 10x as long, but still no error

noisesmith02:06:49

oh I wasn't taking from the timeout!

noisesmith02:06:27

that just changes wait time though

noisesmith02:06:06

oh cool, with enough of those timeouts I was able to get the state machine to start crawling and my CPU is heating up :D

noisesmith02:06:20

anyway, I bet @ghadi would be much more informative on this subject

noisesmith02:06:00

experimentally with enough parked go blocks the throughput gets really low and bursty

kenny02:06:18

Any idea if any of the go blocks gets dropped?

noisesmith02:06:36

the numbers in the atom move slower but everything still looks correct

noisesmith02:06:01

but that's not real proof of anything, it just means my toy test worked without making core.async break

noisesmith02:06:28

I think the assertion error you got was because everything was trying to use c

noisesmith02:06:12

or too many things using some queue that my example doesn't use, clearly

pmooser18:06:22

I am seeing something in the cljs implementation of core.async that looks like a serious bug. It appears to me that using a promise-chan will break non-deterministic choice in cljs. Like, if I have an alts! choosing between a timeout and a fulfilled promise-chan, it appears to never choose the timeout chan at all.

pmooser18:06:44

Something along the lines of:

pmooser18:06:14

(go
  (let [p (promise-chan)
        t (timeout 1000)]
    (>! p {})
    (go-loop []
      (let [[val port] (alts! [p t])]
        (if (= port p)
          (do (println "loop") (recur))
          (println "huh it timed out"))))))

pmooser18:06:33

From looking at the implementation, I don't understand why this would be, since the promise chan is just implemented in terms of a special buffer. We ran into this when porting some code from clj into cljs and started seeing some confusing behaviors (basically infinite loops). I'd be happy to be mistaken here but I'm not sure this is user error.

hiredman18:06:14

I believe that is a consequence of js being single threaded

pmooser18:06:50

I don't understand why, since it has an alts! in the middle, and it's rewritten in terms of being a state machine. This isn't just like a normal tight loop, is it ?

hiredman18:06:26

in order for the timeout to fire the control of the js thread needs to be yielded

pmooser18:06:26

I mean, to be clear, if you replace the promise chan with a separate go-loop that infinitely puts onto a chan, you don't get this same behavior.

pmooser18:06:38

What I am saying is, doesn't alts! yield it?

hiredman18:06:50

basically no

hiredman18:06:05

it can yield it

hiredman18:06:33

but if it can commit to one of the channels without yielding I believe it does so

pmooser18:06:56

That seems like it breaks the entire idea of non-deterministic choice, from a CSP perspective, doesn't it ?

pmooser18:06:25

It seems like it makes it a challenge to write safe code unless you know the provenance and type of every channel you might be alts!-ing on ...

hiredman18:06:50

I am not defending it

pmooser18:06:19

(To be clear, I appreciate your help and insight.)

hiredman18:06:13

but js is a tricky runtime, and I hate it, and as far as I can tell everyone who writes clojurescript(using core.async or otherwise) is basically teeting on a razors edge of fast behavior for the optimal case and falling off some weird cliff for other cases

hiredman19:06:10

but at the same time, if you replaced your random number generator with 5 for non-deterministic choice, I am not sure csp says that is "wrong"

pmooser19:06:11

Yeah. It feels pretty dangerous, to be honest. It sucks having code that runs stably for months in the clj version seemingly to 'randomly' hang on the cljs version for mysterious reasons that are hard to understand.

pmooser19:06:35

I guess it depends on whether we think non-determinism is allowed to mean 'determinism'.

pmooser19:06:03

In any case, it seems to me that if I just stop using promise-chans (or anything else where the buffer might indefinitely supply results), and just use a manually-written go-loop equivalent, the problem won't occur. I hope!

pmooser19:06:27

Hey @hiredman, do you happen to know the code well enough to give me a reference to the place where it will choose not to yield? I'd love to read it (although my understanding of the core.async internals is poor at best).

pmooser19:06:18

In any case, thanks for the help !

hiredman20:06:30

I am actually not sure if it is entirely due to the single threaded nature of js

hiredman20:06:42

alt! is non-deterministicly choose the first available, and in your case unless the timeout has expired, a fulfilled promise will always be chosen

hiredman20:06:45

as for not yielding, the state machine callback is passed to ioc-alts!, which calls do-alts! and if do-alts! doesn't return nil, ioc-alts! returns :recur which continues the state machine loop

hiredman20:06:24

I am less familiar with the cljs side of things though

hiredman20:06:51

My understanding is to some degree informed by https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml which isn't about core.async per se, but has some analogous parts. it describes channel operations as either happening in the optimistic case or the pessimistic case. The core.async code that corresponds to the optimistic case results in no yield to the js thread.

pmooser21:06:19

Yeah, it may be that my understanding is a bit broken as well, because I hate the idea that I have to know what is on the other end of chans (or what types of chans they are) to have an idea of how my code is likely to behave.

pmooser21:06:50

Especially given that chans have no easy mechanism to know their identities or types or anything like that (ie, they do not have names and probably do not implement IMeta). It seems fundamentally more fragile than it needs to be.

hiredman22:06:00

I guess? it still isn't clear to me if you are passing an expired timeout in or not, if you are passing in an unexpired timeout, and a promise-chan that has been delivered to, the fact that it picks the promise-chan every time is 100% everything working as expected and as designed, and if you expect otherwise then you don't understand the kind of choice alts! makes

hiredman22:06:30

alt chooses the first thing that it can(the first operation that can complete), and if it can choose multiple things it picks one at random

hiredman22:06:16

the choosing at random is only done to break a tie, and if one of the things is a take from a timeout channel that isn't expired yet, and the other is an immediately doable take from another channel, there is no tie to break