@drewverlee: That diagram represents a virtual peer, technically - or the lowest level worker.

I often use peer and virtual peer interchangeably because they should be fairly transparent, but more correctly, 1 peer -has many- virtual peers.

Its supposed to emulate the idea of virtual sharding, but how well I actually did that is certainly up for debate.

Thanks @michaeldrogalis and @gardnervickers.

I probably have several fundamental things wrong in my mental model. Is it ever the case that a job (what i think of as a function like (do-something segment) is shared across multiple machines? Or can you only scale up to handle a larger job?

@drewverlee: any task, aside from some plugin exceptions, can run on more than one peer at a time. A peer has one task thread that it'll run an onyx/fn on, allowing you to scale out as you add cores (whether on one machine or more than one machine)

@gardnervickers: would it be fair to think of a job as a topology? I’m forming this intuition that 1 job = 1 topology

(let [job {:workflow c/workflow
                 :catalog catalog
                 :lifecycles lifecycles
                 :windows c/windows
                 :triggers c/triggers
                 :task-scheduler :onyx.task-scheduler/balanced}
            job-id (:job-id (onyx.api/submit-job peer-config job))]

You may have 30 peers running, and 2 jobs that only require 5 peers each.

Then it’s up to the scheduler you choose as to how those jobs are distributed

peer’s are like “workers”, all of the workers at an organization need to coordinate, clock-in and clock-out, communicate with each other even though they might be working on separate tasks.

In the ring example, you can think of it like each worker making sure another worker is still at their table doing what they are supposed to do.

Not sure what you mean by topology, I’d have to see where you’re getting that word from.

michaeldrogalis: Hi, I'm testing out the kafka0.9 plugin. Stumbled upon one issue so far; it's not possible to resume from latest offset between job instances because the task-id is used as part of the checkpoint key

Ahhh yuck. Thanks for finding that one @ckarlsen.

Ill get that patched up tonight/tomorrow. Still needs formal review from others. Did you run into anything else?

Oh right, I’ll review it tomorrow, that was the other thing I needed to get on to

@michaeldrogalis: what I did with datomic’s read-log there is allowed a custom checkpoint key to be supplied. I assume that’d be your fix anyway

i'll let you know if anything else might come up

Cool, yup. I must have backed over that change.

I was able to remember where I was with the onyx-ruby project and make all the outstanding changes I had left on it in one sitting. Leaving a note for myself FTW.

"... and that's why You Always Leave a Note."

Ha!

I'll take one more look over it tomorrow with fresh eyes then get it moved to the onyx-platform organization