So I have a weird situation, where a core.async consumer is pushing messages to some external queue. And this can be a slow operation.

Hence I want to use a pool of consumers to do this.. What is the prefered way of doing this? call that go block in dotimes?

I would prefer if I could perform this in a separate pool which for net/io operations..

@kamd You are talking about blocking io operations here, i.e. you want a pool of real threads?

If so, you can just use thread or thread-call. Calling it in dotimes would work but you'd probably better map over a range instead so that you can keep references to the thread. You'd need those to implement graceful shutdown.

thread-call will use the same pool for go blocks (with 8 as default size) right?

nope

thread-call uses a cachedthreadpool

(unbounded)

I personally tend to use a custom version that accepts an executorservice instead

Hmm is there any way I can provide a custom threadpool, I would like to use a bounded pool..

when I use thread, which is quite rare anyway

it's quite easy via interop, or one of the many wrappers out there for this

ex https://github.com/mpenet/knit

If you call thread / thread-call a fixed number of times, it doesn't really matter that the underlying pool is unbounded, does it

If you want to dispatch tasks (as in: functions) to those threads, then an executor might be more convenient, though

Otherwise you'd have to use a semaphore or something to limit thread

if fixed no it doesn't, cachedthreadpool resizes anyway

it's dangerous in "unbounded" scenarios, ex for every response on your webserver

Indeed

@dergutemoritz thread-call it is then. Thanks

YW! 🙂

anyone know if this is expected behavior in ClojureScript: https://gist.github.com/djwhitt/bd54b89937e13f97f9fb37858b1476ca

if you have a busy loop with alts! and a timeout, the timeout doesn't fire. I would have expected the call to alts! to allow the timeout to fire prior to picking a channel

calling recur in a non-tail position isn't valid

yeah, I'm surprised that code runs at all

you should get an error compiling that

djwhitt: to wait for a timeout, you have to read from its channel

yeah, I realize the recur is funky. don't pay too much attention to that

it isn't funky

it is invalid

t-out (<! (timeout 5000))

that code should not compile

noisesmith: the call to alts! is attempting to read from the t-out channel

hiredman: well... I can try running it again, but I ran a version of it earlier

yeah, the problem is, if it is compiling with a recur not in the tail position, it is so broken who knows what it is doing

I had a version that had recur in tail position, but it was much more complicated (attempting to solve a bigger problem)

I'll rewrite this though, just to be sure...

that would be trivial to translate to cond, rather than multiple when blocks

noisesmith: yep, agreed. I'll do that. I expect the behavior will be the same, but we'll see

please don't pastebin code saying it outputted this that and the other, if the code is not what you ran to get those outputs

hiredman: no, I did run that version

that was distilled from a real world problem I was trying to solve

hah, yep, just redid it because you were making me wonder myself. it compiles without warnings. I'll create a repo with that version and a properly rewritten version

alright, rewritten version: https://gist.github.com/djwhitt/bd54b89937e13f97f9fb37858b1476ca

is that expected behavior?

expected or not, I think it is the only possible behavior, js is single threaded, if you monopolize the thread, nothing else (like firing timeouts) can happen

I thought alts! could be written in such a way that timeouts fired before the channel was chosen

I am not sure what that means

well, alts! itself can block if there is nothing to read from the channels, so I thought that it could be structured such that timeouts were allowed to fire before the channel choice

I could be totally wrong though

that's why I'm asking about it

I guess it could be made to not monopolize the js thread, but performance would suffer because it would yield the thread all the time

this happens because core.async implements its own run queue, outside of the run queue for the js event loop

and timeouts use the js event loop run queue, which core.asyncs run queue starves out

yeah, it's been about 2-3 years since I debugged this, but I'm pretty sure what @hiredman is saying is correct. There are cases in CLJS where you can dominate the run thread because you are putting in a message then reading it out

ah, that makes sense, so not necessarily "expected" per-se, but hard to implement otherwise

so the timeout channel never gets a chance to fire. It's a tradeoff, we could force a "thread switch" on every put/take, but that would result in a performance hit.

gotcha

ok, well, I have a workaround for my code. it's just a lot less elegant, so I was hoping I found a bug 😄

I'm not sure it's a good idea to spin-loop in JS.

the example is contrived. this is actually from a re-frame like event loop. it doesn't normally spin, but if events are firing fast enough it can for a bit

the goal with is actually to make it not spin as much and block occasionally to allow rendering

the timeout was going to serve as a way to detect when the loop had spun for too long

then you may be good, if you have two blocks sending data between eachother, you won't hit this problem

if enough events get queued up it's still a problem

core.async already only processes X messages before switching and allowing the browser to run

IIRC, it's something like 128 msgs.

The problem here is that you are sending and receiving from the same block, and that's a different code path that bypasses that logic

interesting, this may actually improve as I improve the performance of some of my events then

I have a few that are pretty CPU hungry

tbaldridge: oh, that's interesting (re different path)

oh... actually, that happens in the real code too now that I think about it

because events can return other events

that get dispatched in the same loop

maybe I could figure out another way to structure that

anyway, thanks for explaining. that definitely helps me understand the behavior better

tbaldridge: do you want those non-tail-recursive loops compiling written up as a bug?

what version of core.async are you on?

As this doesn't compile for me (a/go-loop [] (do (recur) 1))

and that check is run by tools.analyzer, which is pretty good about catching this sort of thing

0.2.395

oh nvm...this is CLJS...all bets are off then

yeah, that should probably be a bug report

ok, will do. I'll update to the latest too and make sure it's still happening there

@kamd you can also use async/pipeline functions to create consumer with specified parallelism. There are pipeline functions for blocking and acync operations. https://clojuredocs.org/clojure.core.async/pipeline-blocking