Clojurians Log v2

Link? I haven't done a lot of stopping and starting of individual processes, but stopping flows is useful at dev time in the repl, when you need to redefine the flow

I've got a little project that (mis?)uses a flow as the sort of "init" system, sort of replacing something like component, and starting/stopping the flow become like starting and stopping the system

invariably some of the processes in a flow will be sources or sinks, of information external to the flow, and as such might have related resources that need to be created and cleaned up. So that outer edge of the lifecycle is for that. Also ensures that a created but not yet started flow is essentially information.

there’s no API for start/stop process independent of the entire flow, just pause/resume which is an inner lifecycle around the interprocess flow

My intended use case is using flows for processing audio/video streams. A common use case is to have a single input stream feed into multiple filters that get combined later:

/---- filter A ---\
video-stream --<                    >--- combine video streams side by side
                 \---- filter B ---/

We've been using mult such that multiple listeners can subscribe to the same feed of data. In our particular case, orchestrating the sending of updates to multiple clients visiting the same page over web sockets. In this case, b would be nice. I would probably use a queue for use case a.

I have a use-case for distributing work. The overall flow is taking asset pricing updates from a single stream, serializing them to Avro, and pushing them serialized to Kafka. We want the serialization part of this distributed across all cpu cores. But there's a wrinkle: pricing updates for the same asset must be processed in order. e.g we can parallelize serialization of (goog-123, aapl-123, goog-456, aapl-456) -> (goog-123, goog-456), (aapl-123, aapl-456).

I have around to A. B is the model I've had in my mind, but the majority of my process outs end up with a single consumer. I have one left with multiple consumers. If you want the racey consumption of A, I am not sure how else you could get it as things currently stand, but you can emulate B with a process that collects coords a distributes messages to them

(actually, given that flow topologies are static, you don't even need to track coords like that, just add a "fanout" process in the graph)

In an Event Sourcing/CQRS scenario, if a stream contains commands, A could be used to handle them, and if a stream contains events, B could be used to update multiple read-side databases.