Fork me on GitHub
#core-async
<
2017-03-27
>
kamd09:03:07

So I have a weird situation, where a core.async consumer is pushing messages to some external queue. And this can be a slow operation.

kamd09:03:01

Hence I want to use a pool of consumers to do this.. What is the prefered way of doing this? call that go block in dotimes?

kamd09:03:41

I would prefer if I could perform this in a separate pool which for net/io operations..

dergutemoritz09:03:32

@kamd You are talking about blocking io operations here, i.e. you want a pool of real threads?

dergutemoritz10:03:04

If so, you can just use thread or thread-call. Calling it in dotimes would work but you'd probably better map over a range instead so that you can keep references to the thread. You'd need those to implement graceful shutdown.

kamd11:03:52

thread-call will use the same pool for go blocks (with 8 as default size) right?

mpenet11:03:26

thread-call uses a cachedthreadpool

mpenet11:03:32

(unbounded)

mpenet11:03:04

I personally tend to use a custom version that accepts an executorservice instead

kamd11:03:31

Hmm is there any way I can provide a custom threadpool, I would like to use a bounded pool..

mpenet11:03:32

when I use thread, which is quite rare anyway

mpenet11:03:04

it's quite easy via interop, or one of the many wrappers out there for this

dergutemoritz11:03:54

If you call thread / thread-call a fixed number of times, it doesn't really matter that the underlying pool is unbounded, does it

dergutemoritz11:03:55

If you want to dispatch tasks (as in: functions) to those threads, then an executor might be more convenient, though

dergutemoritz11:03:13

Otherwise you'd have to use a semaphore or something to limit thread

mpenet11:03:17

if fixed no it doesn't, cachedthreadpool resizes anyway

mpenet11:03:41

it's dangerous in "unbounded" scenarios, ex for every response on your webserver

kamd12:03:34

@dergutemoritz thread-call it is then. Thanks

dergutemoritz12:03:15

YW! 🙂

djwhitt16:03:02

anyone know if this is expected behavior in ClojureScript: https://gist.github.com/djwhitt/bd54b89937e13f97f9fb37858b1476ca

djwhitt16:03:15

if you have a busy loop with alts! and a timeout, the timeout doesn't fire. I would have expected the call to alts! to allow the timeout to fire prior to picking a channel

hiredman16:03:40

calling recur in a non-tail position isn't valid

tbaldridge16:03:47

yeah, I'm surprised that code runs at all

hiredman16:03:49

you should get an error compiling that

noisesmith16:03:54

djwhitt: to wait for a timeout, you have to read from its channel

djwhitt16:03:25

yeah, I realize the recur is funky. don't pay too much attention to that

hiredman16:03:32

it isn't funky

hiredman16:03:36

it is invalid

noisesmith16:03:38

t-out (<! (timeout 5000))

hiredman16:03:44

that code should not compile

djwhitt16:03:39

noisesmith: the call to alts! is attempting to read from the t-out channel

djwhitt16:03:24

hiredman: well... I can try running it again, but I ran a version of it earlier

hiredman16:03:54

yeah, the problem is, if it is compiling with a recur not in the tail position, it is so broken who knows what it is doing

djwhitt16:03:40

I had a version that had recur in tail position, but it was much more complicated (attempting to solve a bigger problem)

djwhitt16:03:54

I'll rewrite this though, just to be sure...

noisesmith16:03:01

that would be trivial to translate to cond, rather than multiple when blocks

djwhitt16:03:29

noisesmith: yep, agreed. I'll do that. I expect the behavior will be the same, but we'll see

hiredman16:03:39

please don't pastebin code saying it outputted this that and the other, if the code is not what you ran to get those outputs

djwhitt16:03:10

hiredman: no, I did run that version

djwhitt16:03:18

that was distilled from a real world problem I was trying to solve

djwhitt16:03:28

hah, yep, just redid it because you were making me wonder myself. it compiles without warnings. I'll create a repo with that version and a properly rewritten version

djwhitt16:03:22

is that expected behavior?

hiredman16:03:59

expected or not, I think it is the only possible behavior, js is single threaded, if you monopolize the thread, nothing else (like firing timeouts) can happen

djwhitt16:03:33

I thought alts! could be written in such a way that timeouts fired before the channel was chosen

hiredman16:03:41

I am not sure what that means

djwhitt16:03:04

well, alts! itself can block if there is nothing to read from the channels, so I thought that it could be structured such that timeouts were allowed to fire before the channel choice

djwhitt16:03:08

I could be totally wrong though

djwhitt16:03:12

that's why I'm asking about it

hiredman16:03:18

I guess it could be made to not monopolize the js thread, but performance would suffer because it would yield the thread all the time

hiredman16:03:48

this happens because core.async implements its own run queue, outside of the run queue for the js event loop

hiredman16:03:08

and timeouts use the js event loop run queue, which core.asyncs run queue starves out

tbaldridge16:03:44

yeah, it's been about 2-3 years since I debugged this, but I'm pretty sure what @hiredman is saying is correct. There are cases in CLJS where you can dominate the run thread because you are putting in a message then reading it out

djwhitt16:03:09

ah, that makes sense, so not necessarily "expected" per-se, but hard to implement otherwise

tbaldridge16:03:20

so the timeout channel never gets a chance to fire. It's a tradeoff, we could force a "thread switch" on every put/take, but that would result in a performance hit.

djwhitt16:03:15

ok, well, I have a workaround for my code. it's just a lot less elegant, so I was hoping I found a bug 😄

tbaldridge16:03:51

I'm not sure it's a good idea to spin-loop in JS.

djwhitt16:03:53

the example is contrived. this is actually from a re-frame like event loop. it doesn't normally spin, but if events are firing fast enough it can for a bit

djwhitt16:03:03

the goal with is actually to make it not spin as much and block occasionally to allow rendering

djwhitt16:03:05

the timeout was going to serve as a way to detect when the loop had spun for too long

tbaldridge16:03:07

then you may be good, if you have two blocks sending data between eachother, you won't hit this problem

djwhitt16:03:20

if enough events get queued up it's still a problem

tbaldridge16:03:33

core.async already only processes X messages before switching and allowing the browser to run

tbaldridge16:03:42

IIRC, it's something like 128 msgs.

tbaldridge16:03:03

The problem here is that you are sending and receiving from the same block, and that's a different code path that bypasses that logic

djwhitt16:03:17

interesting, this may actually improve as I improve the performance of some of my events then

djwhitt16:03:25

I have a few that are pretty CPU hungry

djwhitt16:03:44

tbaldridge: oh, that's interesting (re different path)

djwhitt17:03:15

oh... actually, that happens in the real code too now that I think about it

djwhitt17:03:20

because events can return other events

djwhitt17:03:29

that get dispatched in the same loop

djwhitt17:03:47

maybe I could figure out another way to structure that

djwhitt17:03:30

anyway, thanks for explaining. that definitely helps me understand the behavior better

djwhitt17:03:03

tbaldridge: do you want those non-tail-recursive loops compiling written up as a bug?

tbaldridge17:03:17

what version of core.async are you on?

tbaldridge17:03:39

As this doesn't compile for me (a/go-loop [] (do (recur) 1))

tbaldridge17:03:02

and that check is run by tools.analyzer, which is pretty good about catching this sort of thing

tbaldridge17:03:17

oh nvm...this is CLJS...all bets are off then

tbaldridge17:03:28

yeah, that should probably be a bug report

djwhitt17:03:01

ok, will do. I'll update to the latest too and make sure it's still happening there

serioga17:03:41

@kamd you can also use async/pipeline functions to create consumer with specified parallelism. There are pipeline functions for blocking and acync operations. https://clojuredocs.org/clojure.core.async/pipeline-blocking

dergutemoritz20:03:36

Not sure about pipeline-blocking, but for pipeline-async the parallelism parameter isn't precise (see http://dev.clojure.org/jira/browse/ASYNC-163) so you might want to check that if you're going to rely on it