Clojurians Log v2
core-async
Hi. I've just tested core.async.flow and I quite like it. I've read the guide and reference pages, watched Alex' youtube walkthrough and played around with the example repo. However, I haven't found a way to run a specific Flow process in parallel. Is there a way for a proc to compute as many incoming channel values as possible? In my use case, I need to maximize CPU utilization by shelling out to an external (CPU intensive) CLI tool for each input value and I thought I could control that by providing different input channel sizes for this particular proc. A quick disclaimer, I don't have much prior experience with core.async and I'm still wrapping my head around workload types and underlying executors.
Parallel processing is not going to be "in" the flow, but some external thing that the flow feeds values into, and may consume values from
So a process that feeds values into an executor or something, and then those tasks send values back into the flow via an in-ports channel or something
We've already built a system that parallelizes jobs based on input values with traditional fixed-size thread pools via Claypoole. However, I feel many parts of the system can be simplified considerably with the Flow (CSP) approach.
I see. Then there should be a mix of flow with an external values producer, which should be running in parallel
That method is entirely workable with the standard process stuff in flow
If you are willing to implement your own ProcLauncher you can do whatever
Got it. Then the standard process flow is always sequential. Yes, I'd look into ProcLauncher next, I just wanted to make sure I understand what is already available in the standard model. Thanks!
https://clojurians.slack.com/archives/C05423W6H/p1746639626781339