Hello! What is the usual way or common way people build async workers? Let's say you have a monolith and you need to spawn, at regular times, or based on events, workers to do specific jobs. Usually this comes with a persistence layer like redis to maintain jobs that weren't finished if the server restarts
> at regular times, or based on events
Sounds like a perfect fit for a core.async go block with an alt! over a timeout and a channel that receives events.
> Usually this comes with a persistence layer
That would just mean that something like that go block put stuff onto this layer, and some other process (in the conceptual way, not in the "an OS process" sense) monitors it and executes stuff from it. So, somewhat orthogonal to the "timeout and events" part - "somewhat" because of the part below.
That leaves the bit of run time between something triggering an event and the go block putting it onto the persistent layer. If you care about this gap, the event itself could be delivered via such a persistent layer. Easy to do if you don't need the "exactly once" guarantee. Otherwise, core.async is, I think, a wrong thing to use here.
Found this post which is similar to your suggestion: https://blog.janetacarr.com/creating-a-dead-simple-asynchronous-job-system-in-clojure/ I also found https://github.com/msolli/proletarian which is tied to SQL DBs but still is nice
#caveman also uses proletarian in case you want to see an example: https://caveman.mccue.dev/tutorial/clojure/23_run_jobs_in_the_background
Thank you, that’s very helpful!
Does anybody know of example job “systems” using carmine? My setup uses datomic so I’m not SQL based
I used Carmine's worker stuff in the past, it worked fine
There's also https://github.com/nilenso/goose And you can also use Quartz https://github.com/michaelklishin/quartzite Goose can be Reddis backed.
I use carmine + redis. https://github.com/danielsz/system/blob/master/src/system/components/redis_pubsub.clj is the pub sub component for system.