Fork me on GitHub
#core-async
<
2017-03-17
>
wei15:03:49

is there a way to increase a queue size above 1024? could I make it 16,384?

noisesmith15:03:19

@wei you can pass a buffer object as an argument to a chan, I'd expect you could make your own buffer implementation, but I'd also check why the limit is 1024... see for example this discussion http://dev.clojure.org/jira/browse/ASYNC-23

noisesmith15:03:35

@wei in fact, the patch submitted on that ticket might be a good starting place if you wanted to implement your own buffer

hiredman15:03:02

if you are hitting the 1024 limit you might want to re-examine how you are handling pack pressure too

wei15:03:14

is there a good resource on back pressure?

wei15:03:28

let’s say I don’t want to drop anything

hiredman15:03:30

are you using put!?

hiredman15:03:31

basically, for back pressure to work, you can't do thinks as fire and forget

hiredman15:03:51

if you have a loop that spins off go blocks, the loop needs to wait for those go blocks to complete (that would be the simplest way, you can get feeback other ways too)

mpenet15:03:12

@hiredman you actually can use put! callback + on-caller? to signal backpressure upstream, as long as you're ok accepting to have 1 message queued in the case of block

mpenet15:03:34

not that it's the best thing to do tho, It's best to use >! <!

joshjones15:03:37

@wei if you're using >! (as you said), then the 1024 limit shouldn't really matter to you, as that applies to non-blocking puts made by put!. >! will park if the buffer on the channel (not the same as the puts queue) is full.

hiredman16:03:10

mpenet: I agree it is simpler to use >!,`<!`, <!!, and >!!, a beginner is much more likely to get a system that doesn't have backpressure problems that way. Some people and documentation steers people towards using put!, which I think is much more likely to cause a lack of back pressure, which is why I asked about it

mpenet16:03:36

yes, put! is usually a smell

joshjones16:03:12

@mpenet usually using put! is preferred to simply wrapping >! in a go for the sake of parking, as the go block is not free. I've done what you said, namely, use the put! callback to ensure that the put queue does not grow.

mpenet16:03:08

also the put queue is per chan, using >! doesnt' guarantee anything

mpenet16:03:23

(def x (a/chan))
(dotimes [i 1025] (a/go (a/>! x i)))
would blow up

hiredman16:03:55

things are a little different on the clojurescript side of things, but on clojure I would use >!! (the real thread blocking one outside of a go block) over put!

joshjones16:03:12

in certain situations, perhaps -- but in an http async request handler which is going to put the result on a channel, >!! is (unnecessary?), as one example

wei16:03:09

thanks for the resources and tips. will need to review our design

wei16:03:47

in our particular case the problem ended up being a single user spamming the system. so rate-limiting that user solved the problem for now!

joshjones16:03:09

pesky users 😉

wei16:03:30

we were using >! and getting the 1024 error though

joshjones16:03:46

yeah, I guess it was a case similar to the example @mpenet posted above -- but curious, what is the size of the buffer on the channel, if it's buffered?

mpenet16:03:22

the size of the buffer is what you ask it to be. The pending put! queue is another thing, and it's always 1024

mpenet16:03:42

nothing prevents you from create a chan with a giant backing buffer

joshjones16:03:45

yeah, i'm asking, what's the size of his buffer

mpenet16:03:56

(chan) is unbuffered

mpenet16:03:04

(chan n) is n sized

joshjones16:03:57

didn't I just say that? what does that have to do with the size of wei's channel's buffer?

mpenet16:03:35

not sure I follow, but the 1024 on the pending put! queue is hardcoded I think

joshjones16:03:06

To be clear, I'm not asking you anything about this -- I'm asking @wei, if his channel is buffered, and if so, what is the size of the buffer on his channel -- unless you looked at his code, and you know how large the buffer is, on his channel 😉

wei16:03:31

it’s unbuffered

mpenet16:03:46

oh, well didn't realize you were addressing him directly

joshjones16:03:22

@wei any reason you can't buffer it? -- it may not solve your problem in this case, but curious if it may give you some breathing room for cases like this

wei17:03:43

true, that might have reduced some stress. although in this case it was helpful to see the error so we were alerted to a problem

tbaldridge17:03:44

btw, (go (>! c val)) is also a code smell...that pattern is exactly the same as (put! c val) except slower

wei17:03:04

good to know, thanks Tim. I was wondering how those differed

tbaldridge17:03:34

they both attempt to use a channel channel without directly respecting backpressure. So if you're getting that "1024" limit...it should be considered a bug, and a suggestion to refactor your code.

tbaldridge17:03:23

(I said "directly" since you can make put! respect backpressure, but it requires the programmer to worry about the mechanics of it)

wei19:03:06

I’m seeing an error where (<! c) is returning the same value over and over, seemingly without anything additional being enqueued. has anyone else seen this as a failure case? trying to get an isolated repro

noisesmith19:03:55

other than nil of course, no

mpenet20:03:52

Could be a promise-chan, or a chan with a weird xform

bbrinck22:03:12

My intuition is that after completing a test that includes a go block in CLJS, cljs.core.async.impl.dispatch/running? would be false and cljs.core.async.impl.dispatch/queued? would be false. But when I print out this information after running a test (using an :each fixture), I see the opposite (they are both true). From reading over the source, I find that surprising, since these lines https://github.com/clojure/core.async/blob/f8e87e1625b1660b7f3b0aea044aad1327441741/src/main/clojure/cljs/core/async/impl/dispatch.cljs#L14-L15 show that as soon as we set running? to true, we set queued? to false

bbrinck22:03:01

The test I am running is this test (https://github.com/clojure/core.async/blob/master/src/test/cljs/cljs/core/async/tests.cljs#L42-L46) copied over into my project for convenience

bbrinck22:03:01

Is this behavior expected?

bbrinck22:03:48

The reason I ask is that I’m attempting to verify that running tests doesn’t create an unfinished core.async work that might be slowing down my tests

tbaldridge22:03:27

that's internal implementations that really shouldn't be messed with in user code

tbaldridge22:03:37

but the idea is this (since I wrote that code):

tbaldridge22:03:19

We used to simply call setTimeout whenever we needed to call a callback "as soon as possible" but this introduced some lag. So that code now works in two-phases.

tbaldridge22:03:44

1) queued says "we have executed setTimeout and it will run sometime in the future when the browser has some spare cycles

tbaldridge22:03:55

2) runnng? says "we are currently running that stack of pending callbacks"

bbrinck22:03:33

that makes sense. And you’re right, we should not depend on the internals. This is just some debugging I’m doing to try to understand our code and tests.

bbrinck22:03:02

With your explanation above, it seems weird that after a test, both running? and queued? are true though, yes?

bbrinck22:03:38

because it looks like as soon as we start running, we stop being queued

bbrinck22:03:43

FWIW, I’m also inspecting the length of the tasks buffer. That may be more useful than looking at running? and queued?. For instance, if I write a simple test that just starts a go block but incorrectly does not use the async macro, I can see that there are tasks left in the buffer.

bbrinck22:03:13

Nonetheless, I was surprised to see both running? and queued? set to true, but that’s likely because I’m misunderstanding something about the implementation

bbrinck22:03:51

(my suspicion is that somewhere in our test suite, we do in fact have a test that is using core.async incorrectly, so I’m trying to look into the internals of core.async to determine if any test leaves things in a bad state)

tbaldridge22:03:32

well if the code you are checking the status in is in a go block, then you will always be running?

bbrinck22:03:11

I believe this should be outside that block (it’s in a fixture). Let me post some code (do you prefer in slack or a gist?)

bbrinck22:03:21

(thanks for the help, btw!!! much appreciated)

bbrinck22:03:50

“outside the go block” I mean

tbaldridge22:03:59

gist is fine

tbaldridge23:03:33

it looks like you're calling (done) inside the go

tbaldridge23:03:26

if you did something like (js/setTimeout (fn [] (done)) 1) you probably see what you expect

bbrinck23:03:22

I will try that. I took that test from https://github.com/clojure/core.async/blob/master/src/test/cljs/cljs/core/async/tests.cljs#L42-L46 (although I see that I left my unnecessary close! call in my example by mistake)

bbrinck23:03:24

You’re right, that worked! Hm I wonder if I could also fix using take! on the go block, or if I’d run into the same problem

bbrinck23:03:27

Nope, take! didn’t work, but your setTimeout idea did.

bbrinck23:03:22

So, I’m guess the issue is that we call done while we’re still running core.async code, but done signals the test can end and therefore the fixture is called

bbrinck23:03:06

the setTimeout works because it just waits for a bit before signaling the test to end, and at that point, core.async isn’t processing work anymore

bbrinck23:03:52

Thanks for the help! We have another fixture in our tests and I wonder if this behavior isn’t causing problems there. I will check it out!