Is thread-call intended to be called with any workload other than the ones core.async provides itself? (e.g. :mixed, :io, etc…) Let’s say I want a fixed size thread pool to match a specific number of connections to an external service, could I pass e.g. :database as a workload to thread-call and then have that matched with code in my custom “clojure.core.async.executor-factory” to provide just the right sized thread pool for that one use-case?
I’m just working on utilizing this exact tactic to manage multiple bounded compute pools so I’ll be interested in replies
It looks very doable - I just don't want to be working against the intended design and ending up in an unsupported situation.
same
I did find a few places where I wished core.async would provide a public/non-impl way to do something or make something pluggable. I ended up depending on a few impl details and needing to reimplement some others I didn’t want to depend on or wanted to extend.
no, it is not intended to add arbitrary thread pools to it, it’s intended to support specifically the thread pools in core.async
that’s interesting, given the current extension point (executor factory) needs to reimplement threadpool selection based on workload
could it support more? I suppose, but I would recommend using qualified keys that you control for the pool names
If we namespaced our keys would that then count as a reasonable use of the library, or would we be rolling the dice with regard to future changes do you think?
we were not intending to create an open facility so I think you’re outside what we were trying to provide or what is documented, but I don’t think it’s a crazy thing to do and I think it’s unlikely to change much soon.
Is there a better way to specify particular thread pools for particular tasks using core.async?
(feel free to tell me to go read the docs if I’ve missed something obvious)
(I suppose I can still use channels and their put/take operations, while opting out of using thread-call and things that build on it to create my [virtual?]threads.)
Why do you need particular thread pools? All virtual threads are effectively served from the same jvm pool so there is no reason not to just use :io for that
In my case the workloads are different categories of computation (for example: workload 1 - http response coercion vs workload 2 - invoking a complex calculation somewhere else) which currently either run directly in a go block or an async/thread. I’m introducing dedicated pools for these to isolate them
Let’s say I have one group of operations reading data from S3 and another writing to a database - ideally the S3 operations would be limited to <= the number of connections in the S3 connection pool and the number of DB operations limited to <= the number of DB connections. This way I avoid failures where I am unable to acquire connections from connection pools. Ideally, those two limits would be specified separately (as they are not inherently tied together).
An alternative is to use a separate concurrency mechanism (like a semaphore) to limit the parallelism of those operations - but ideally I would avoid adding extra mechanics when core.async is already in play.
(I hope that makes sense.)
there is very little benefit to creating multiple other pools for this use imo
One thing that separate pools can give you is separate monitoring of contention for example.
In my case, I do have more connections in the pool than cores, so yes, that’s probably fine. Thank you. gratitude
It also occurs to me (after discussion with my colleagues) that it might be better in my case to create code that blocks when connections are unavailable from pools, rather than trying to limit the number io threads.