clojuregeek: I think kafka could work for that kind of monitoring / status streaming

clojuregeek: I think it's a good fit because it is essentially a logging system that got turned into a message queue

yeah that has crossed my mind..

in addition to monitor a bunch of aws machines, if certain conditions are right (maybe one is running out of diskspace) I want to do certain actions, like expand the hdd or add more storage

with kafka… i could monitor and if conditions are right to expand hdd .. then maybe i have another topic where I post a message when the disk drive needs to be expanded…

The appealing surface aspect of Kafka is its durable, total ordering of messages - which is incidentally the cornerstone on which most workflow engines are built. The danger of going the Kafka-route is that you stick yourself with all the problems of fault tolerant coordination, which get especially tricky when there are multiple days between messages or signals.

Just my 2 cents, anyhow.

😕 so … still on the drawing board..

just using SQS or other messaging queue is not enough to deal with async execution of the worker tasks?

at least in theory you could even use Lambda to execute those tasks (as you say, they are pretty small and focused)

protip: stick theory tag on my suggestion, since I have no idea about how Lambda works beyond the marketing material 🍷

andrewhr: clojuregeek was talking about things that weren't strictly tasks pushed to workers (eg. monitoring stuff) which is why I suggested something outside the world of onyx proper

and of course onyx would handle the doing and dispatching tasks aspect nicely, but that's only one part of what I though I saw described