@michaeldrogalis: count me in! 🙂

@bcambel: Awesome. 🙂

We'll send a poll out shortly.

noob question. Can a input stream into onyx be routed to distributed peers, where each peer is trying to do the same task? e.g read -> some-task -> output with 2 servers A and B. So

input stream: msg1, msg2…

server A ends up with msg1 and does some-task(msg1)
server B ends up with msg2 and does some-task(msg2)

Currently we only allow one aggregation per sequential tasks in a job, you cannot put your aggregation result back into the workflow and pass it on to other tasks.

But we do support multiple some-task’s running on different peers with their own windows, these windows will merge into one logical window.

The resulting aggregation can be :trigger/sync’d to durable storage, but cannot be passed to output

The peers do not need to coordinate in the time domain because window’s are a result of the :event-time key on a segment. Each window just buckets based on the time range that the segment falls in, based on it’s :event-time

@gardnervickers: Thanks! I have about 50 follow up questions but ill limit it to three on what you said here: > But we do support multiple some-task’s running on different peers with their own windows, these windows will merge into one logical window 1. Is a task running on multiple peers the default mode of operation? 2. Does the user (like me) have to provide any information to onyx for it to run a task on multiple peers? 3. So assuming the peers each have their own window, does the merge happen at the trigger? Or does it happen as msgs come in. For example if at 1:00pm server A gets msg1 and at 2:00pm server B gets msg2 and the trigger happens at 3:00pm are serverA and B exchanging information at 3:00 or do they do it earlier?

1. It depends on your choice of task scheduler https://github.com/onyx-platform/onyx/blob/6d838c9d3a70663d0f84dbbd3021acc54393d009/doc/user-guide/scheduling.md#task-schedulers 2. Depending on your choice of task scheduler, you might have to set upper or lower limits with :onyx/max-peers or :onyx/min-peers in your catalog entry. 3. As a user you wont have to worry about this, when your trigger fire’s you’ll get the entire logical (cluster-wide) window state.

2. Is also affected by what flux policy you choose, for example if you’re doing a grouping on the stream so that every segment with key > 5 goes to Node A, and every segment with key < 5 goes to Node B, it would not make sense to scale up in the middle of a job, or continue running a job if Node B goes down.

@drewverlee: Let me know if anything was unclear, this stuff is not easy 😄

@gardnervickers: Thanks, I think understand what you said. However its possible i have a mis-undestanding on what a job is, as this description from the docs confuses me a bit... > A job is the collection of a workflow, catalog, flow conditions, lifecycles, and execution parameters. A job is most coarse unit of work, and every task is associated with exactly one job - hence a peer can only be working at most one job at any given time. I would expect a virtual peer to be only able to work on one “task” e.g (split-words or count-words, etc…). By saying the peer works on a job, and that a job contains a workflow, it almost feels like the same peer is going to do both tasks (split-words and count-words). So in a workflow like [:do-A -> :do-B i] with a greedy scheduler i would expect all nodes to try and :do-A first then once do-A was done to do-B. The docs suggest that a job contains the workflow … which i don’t quite grok

Yea so a peer can only be involved in a single job. That peer might be concerned with lifecycles/windows/triggers that are part of that job.

Further, a peer can only be involved in a single catalog task in that job. Again, it might have multiple lifecycles, windows triggers, but it can only be involved in one task.

ok, that makes sense. so with a workflow like :do-A :do-B and a greedy scheduler then all peers will :do-A tell it has no more traffic then try to :do-B?

If both :do-A and :do-B dont have any restrictions then half the availible peers will :do-A and the other half with :do-B

Think of it like this, each catalog entry is a template for some kind of transformation, windows/trigger/lifecycles are attached to these templates. Peers pick up a template w/ attached windows/triggers/lifecycles and run them.

Flow conditions link different templates together in a special way

so a job can have many tasks? Then if a peer can work on 1 job, it can still work on many tasks. Does that mean a peer will work on tasks without sending the msgs to other peers? so in my a simple example. If we have a job with a workflow like [:split-words, :count-words]. Then when this peer receives a msg {:sentence “I love cats”} it will split them, then count them then send the outputforward? The alternative in my mind is that it splits them {words: [I, love, cat]} and sends that pair onto the wire for another peer to pickup. In the first way, the workflow is done all on a single peer, in the second the peer only handles a single task.

No, a peer can work on one task at a time. However, if there are two jobs that have the same task “template” the peer will be bound to one job.

`

Peers are seperate from nodes, right?

so you can have 100 peers on a single machine if you want

A peer represents one piece of a job, this is usually a transformation like :split-words or something. This piece is called a task

A job is made up of many such transformations

When a peer decides it wants to participate in a job, it claims one of the tasks in the job

It cannot participate in any other jobs, even if the task is exactly the same in another companion job

so if a peer is dedicated to a task, does that mean all it does is split-words until the job is finished? Like if there is an unbounded set of data, then a peer on a node will be assigned split-words forever? i should probably dig into: http://www.onyxplatform.org/docs/user-guide/latest/scheduling.html more...

Yea it’s not going to stop splitting words and switch to another task

I see where you might be confused on it, like if a task splits a bunch of words, then quickly switches context to count a bunch of words or something

Onyx does not do that

Depends on which scheduler you use. If you use a balanced job scheduler and add more jobs, Onyx will try to accommodate and move some peers to new tasks so all jobs can operate.

The doc you linked to will answer most of your questions about this topic I think. Happy to answer if you're still confused after reading that.

But if you had 3 peers and a workflow like :in -> :inc -> :out, the peer assigned to :in wont ever jump over and start :inc’ing segments

@michaeldrogalis: @gardnervickers thanks, ill did into the docs a bit. I wasn’t even sure what i was unsure about when we started 🙂.

No prob 🙂

Nice, yea this stuff takes a bit to gain a mental model for