This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-04-20
Channels
- # beginners (35)
- # boot (81)
- # braveandtrue (7)
- # cider (74)
- # cljs-dev (52)
- # cljsrn (5)
- # clojure (114)
- # clojure-austin (1)
- # clojure-belgium (3)
- # clojure-brasil (8)
- # clojure-czech (7)
- # clojure-greece (18)
- # clojure-ireland (1)
- # clojure-russia (67)
- # clojure-uk (11)
- # clojurescript (52)
- # core-async (5)
- # core-matrix (3)
- # cursive (13)
- # data-science (3)
- # datomic (8)
- # emacs (80)
- # events (2)
- # hoplon (8)
- # immutant (7)
- # jobs-discuss (3)
- # juxt (25)
- # ldnproclodo (21)
- # leiningen (27)
- # mount (24)
- # off-topic (6)
- # om (23)
- # onyx (48)
- # parinfer (1)
- # proton (4)
- # re-frame (41)
- # reagent (9)
- # spacemacs (8)
- # untangled (122)
just upgrading to 0.9... i see that :onyx/restart-pred-fn
is deprecated in favour of :lifecycle/handle-exception
- one question - i had a (Thread/sleep 5000)
in my old :onyx/restart-pred-fn
to prevent the thrashing that otherwise happens when kafka brokers get restarted - is that still a reasonable thing to do ?
Another curiosity, thinking more about peers in a cluster - what exactly is pushed between peers? What do peers need to share in order to confidently participate in a job?
I've had an idea about dynamically creating workflows and catalogs and submitting them as jobs - would these be able to scale?
Peer’s communicate through the zookeeper log, not directly. They race to try and participate in a job using zookeeper. The entire job datastructure is stored in zookeeper and downloaded by the peers.
onyx.api/submit-job
just writes to zookeeper, it does not talk to the peers directly.
Any time you want to change code you need to redeploy.
That’s just pure data, you can manipulate and re-submit the job/workflow
So we could create novel workflows and catalogs so long as the catalogs reference functions that can be resolved by the peer?
Exactly
This leads to a desire to keep your functions somewhat general and parameterize through the catalog.
i.e. instead of a catalog entry that’s :onyx/name :inc-5
you would have
{:onyx/name :adder
:onyx/params [5]}
This snippet might be helpful, this is how onyx turns a keyword reference to a function into a function https://github.com/onyx-platform/onyx/blob/686ca2a8bb1fed8cd3a6dbe63d75923460f46888/src/onyx/peer/operation.clj#L27-L34
That applies not only to catalog functions, but lifecycle functions, trigger :sync
functions, etc…
as long as it can be resolved by that, you dont need to redeploy your jar
i think it got weird when i started seeing that the catalog was resubmitted each time
Slight tanget, is there a preferable way to submit a job to a cluster? Just contact one peer and submit-job
?
submit-job
just talks to zookeeper, you dont need to run it on a peer.
It just needs the job datastructure
and uses the env-config
to resolve zookeeper
@tcoupland: I’m not sure what your talking about, where were you seeing the catalog being resubmitted each time?
(let [job {:workflow looped-flow
:catalog dev-catalog
:lifecycles dev-lifecycles
:flow-conditions flow-conditions
:task-scheduler :onyx.task-scheduler/balanced}]
(onyx.api/submit-job peer-config job)
@mccraigmccraig: yeah, it should work pretty much identically, so you can do what you did before
@tcoupland: Oh I see, that’s a common pattern among Onyx users. You have two entrypoints to your Uberjar, one that starts up the peer and one that submits the job. That makes it so you dont have to do anything extra during deployment except start your jar with a different entrypoint.
@gardnervickers: i think we're sorted now. Thanks, was a silly question really, there's no way it would have been made the other way, but sometimes you've just got to know
thanks for your help @gardnervickers
No problem! It’s totally not a silly question, Onyx has a different architecture than most are used too.
"so, the peers essentially just contain a bunch of functions .. then you submit 'job' explaining them and how to wired them together .. for a given purpose" - @tcoupland This is a great simple explanation.
@michaeldrogalis: thanks! feel free to use it