This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-05-02
Channels
- # admin-announcements (4)
- # aleph (10)
- # arachne (1)
- # beginners (66)
- # boot (19)
- # cider (6)
- # cljs-edn (2)
- # cljs-site (32)
- # cljsjs (4)
- # cljsrn (32)
- # clojure (116)
- # clojure-austin (6)
- # clojure-belgium (2)
- # clojure-dusseldorf (1)
- # clojure-russia (16)
- # clojure-uk (5)
- # clojurescript (178)
- # community-development (2)
- # cursive (28)
- # datascript (16)
- # datomic (16)
- # dirac (13)
- # editors (2)
- # emacs (1)
- # error-message-catalog (30)
- # events (3)
- # garden (2)
- # hoplon (27)
- # jobs (4)
- # ldnclj (4)
- # liberator (3)
- # off-topic (6)
- # om (49)
- # onyx (24)
- # parinfer (9)
- # re-frame (59)
- # reagent (46)
- # remote-jobs (1)
- # rethinkdb (4)
- # rum (2)
- # slack-help (11)
- # untangled (13)
@drewverlee: That diagram represents a virtual peer, technically - or the lowest level worker.
I often use peer and virtual peer interchangeably because they should be fairly transparent, but more correctly, 1 peer -has many- virtual peers.
Its supposed to emulate the idea of virtual sharding, but how well I actually did that is certainly up for debate.
Thanks @michaeldrogalis and @gardnervickers.
I probably have several fundamental things wrong in my mental model. Is it ever the case that a job (what i think of as a function like (do-something segment)
is shared across multiple machines? Or can you only scale up to handle a larger job?
@drewverlee: any task, aside from some plugin exceptions, can run on more than one peer at a time. A peer has one task thread that it'll run an onyx/fn on, allowing you to scale out as you add cores (whether on one machine or more than one machine)
@gardnervickers: would it be fair to think of a job as a topology? I’m forming this intuition that 1 job = 1 topology
(let [job {:workflow c/workflow
:catalog catalog
:lifecycles lifecycles
:windows c/windows
:triggers c/triggers
:task-scheduler :onyx.task-scheduler/balanced}
job-id (:job-id (onyx.api/submit-job peer-config job))]
However, this
> Within that ring each peer tries to claim and run jobs
indicates a ring of peers could be working on 1 or more separate topologies. Which would seem un-necessary.You may have 30 peers running, and 2 jobs that only require 5 peers each.
Then it’s up to the scheduler you choose as to how those jobs are distributed
peer’s are like “workers”, all of the workers at an organization need to coordinate, clock-in and clock-out, communicate with each other even though they might be working on separate tasks.
In the ring example, you can think of it like each worker making sure another worker is still at their table doing what they are supposed to do.
Not sure what you mean by topology, I’d have to see where you’re getting that word from.
michaeldrogalis: Hi, I'm testing out the kafka0.9 plugin. Stumbled upon one issue so far; it's not possible to resume from latest offset between job instances because the task-id is used as part of the checkpoint key
Ahhh yuck. Thanks for finding that one @ckarlsen.
Ill get that patched up tonight/tomorrow. Still needs formal review from others. Did you run into anything else?
Oh right, I’ll review it tomorrow, that was the other thing I needed to get on to
@michaeldrogalis: what I did with datomic’s read-log there is allowed a custom checkpoint key to be supplied. I assume that’d be your fix anyway
Cool, yup. I must have backed over that change.