Fork me on GitHub

Going through the user guide, and the sample project, May I test few assertions about the architecture here? 1 existing cluster deploy tools should be used; no need for Onyx to invent its own mechanism for it 2 peers are network nodes or the process that managers virtual peers on a node, virtual peers are workers running within peers 3 tasks are split evenly across all peers and virtual peers 4 nodes can be added and removed by will, will join the cluster and seamlessly begin performing tasks 5 tasks have types: input, function (which is really the processing part of it all), output 6 number of virtual peers to participate in a task is determined by the max-peers key of the task 7 system-wide snapshots are somehow taken (in the latest version through an algorithm optimizing for space efficiency not so much latency) such that if a node is removed or stops responding, work is never (?) lost. 8 exactly-once semantics are enforced (but side-effects may occur more than once) 9 kind of a rough edge, something to automate in future versions, or just a tuning task for the system architects of a particular system, gc should be invoked by user code or an admin command.


1. Yes. 2. Yes. 3. Depends on the scheduler being used. Controlled by both job and task schedulers, and by per task constraints like onyx/min-peers / max peers. 4. Yes. 5. Yes. 6. max-peers is the upper bound, there are other factors — it can receive less unless min-peers is specified. Otherwise could get some number inbetween. 7. Other way around — sort of. I’d argue it’s generally optimized for latency. We used a more space efficient algorithm in 0.9 — 0.10 dumps an entire snapshot at every epoch checkpoint. 8. Yes. “exactly-once aggregations” are supported. exactly-once side effects are snake oil. 🙂 9. Yeah. Though, most of the time we switch to a new tenancy before invoking GC.


michaeldrogalis: thanks a lot 🙂 I'll proceed my learnings through further reading beyond this now.


@U050A65BL What had you literally meant by "most of the time we switch to a new tenancy before invoking GC"? does does switching a tenancy mean?


@matan Ill get back to you on the mailing list tonight btw


michaeldrogalis: no rush!