Fork me on GitHub

I’m writing a event emitter to simulate website traffic using core.async, and since I’m aiming at ~100k concurrent sessions I’ve been running into!topic/clojure/OSwKOKEupGc


what is the best way to work around this limitation?


at the moment I’m considering to implement some sort of barrier around the timeout chan or to just fork core.async and modify the limit


I also want to point out that this is something I’m writing for fun, so it’s not like it’s blocking some sort of critical production system somewhere, but I’m still interested in finding a solution 🙂


core.async is unlikely to be good for that


core.async is limited, by default, to running something like 8 go blocks concurrently


you could use async/thead and Thread/sleep


it is, but if you want a thread per go block, it seems like, maybe just use threads?


(I am just assuming the reason you would do that doseq is because you want to spin up a bunch of threads to test concurrent connections)


you misunderstand me, I’m not testing a web server and concurrent connections. I’m writing a library to simulate events, such as pageviews, through state machines


and emit those events on a core async channel


I intend to use it to generate test data for various stream processing applications


reading back I see that was less than clear from my initial explanation 🙂


right - I'd say core.async might be a bad fit here because core.async isn't actually good at massive parallelization (which is a pre-req here I think) - it's good at async, coordinating things that are not synchronous, but isn't really a mass scale parallelization lib


though if you use something else for parallelizing on a mass scale then realize you are having trouble coordinating between those threads - then core.async can help again


that’s unfortunate… seemed like a great fit to me since I want to have a lot of sessions but most of them will be idle waiting to emit an event at any given time


seems like every time I find a problem where I think “oh, I can use core.async for this!“, the answer is “don’t use core.async for that” 😕


seems like with a prio-queue you could avoid needing so many state machines by having a queue of "next connection plus its callback"


and, with some fiddling you could make that work with a channel


but directly using that many events in flight is where the problems happen


@schmee core.async can handle that many events, but it has that error because with intended usage having that many things "in flight" indicates a design problem


meh, I don’t like when libraries impose arbitrary restrictions like that. I’m a grown man, I can take responsibility for my own stupidity 😄

Alex Miller (Clojure team)21:01:12

it is not obvious to me that core.async wouldn’t work for this


formulated in core.async terms, my current idea is to have ~100k go blocks, where at any moment most of them are parked on a timeout channel, which each outputs events into a main event channel

Alex Miller (Clojure team)21:01:34

but I have not read all the backchannel so might be missing something

Alex Miller (Clojure team)21:01:03

another option may be agents, which were designed for simulation


I’ll take a closer look at agents and see if that might be a good fit 👍


yeah, I think agents would avoid some of thse issues - the specific reason I think this isn't good for core.async is wanting that many events waiting on timeouts at once (which might be a bug that is fixed in a future core.async but might be just not how core.async is meant to be used?)

Alex Miller (Clojure team)21:01:33

they are just a bucket of state that you can enqueue functions to apply to it

Alex Miller (Clojure team)21:01:56

but they are aware of and able to send functions to other agents


goes to the bookshelf to grab Joy of Clojure

Alex Miller (Clojure team)21:01:28

and the application is backed by a thread pool

Alex Miller (Clojure team)21:01:10

if you send-off, that pool is unlimited in size, but you can monkey with it


I thought send was limited and send-off was resizing

Alex Miller (Clojure team)21:01:03

yes it will resize without limit :)

Alex Miller (Clojure team)21:01:16

but you can change these with set-agent-send-off-executor! etc


alright, I have some new things to test, thanks everyone for your input and time 👍