just upgrading to 0.9... i see that :onyx/restart-pred-fn is deprecated in favour of :lifecycle/handle-exception - one question - i had a (Thread/sleep 5000) in my old :onyx/restart-pred-fn to prevent the thrashing that otherwise happens when kafka brokers get restarted - is that still a reasonable thing to do ?

Another curiosity, thinking more about peers in a cluster - what exactly is pushed between peers? What do peers need to share in order to confidently participate in a job?

I've had an idea about dynamically creating workflows and catalogs and submitting them as jobs - would these be able to scale?

Or does each peer need to have a catalog or workflow itself before participating?

Peer’s communicate through the zookeeper log, not directly. They race to try and participate in a job using zookeeper. The entire job datastructure is stored in zookeeper and downloaded by the peers.

onyx.api/submit-job just writes to zookeeper, it does not talk to the peers directly.

but, in order to run a 'novel' workflow you'd need to deploy another jar?

Any time you want to change code you need to redeploy.

does code include the workflow in that sentence?

No

That’s just pure data, you can manipulate and re-submit the job/workflow

So we could create novel workflows and catalogs so long as the catalogs reference functions that can be resolved by the peer?

Exactly

Bingo

This leads to a desire to keep your functions somewhat general and parameterize through the catalog.

Yes

it was just getting a little confusing when reading around the adding peer docs

i.e. instead of a catalog entry that’s :onyx/name :inc-5 you would have

{:onyx/name :adder 
:onyx/params [5]}

cool, that's how i thought it worked before i confused myself

This snippet might be helpful, this is how onyx turns a keyword reference to a function into a function https://github.com/onyx-platform/onyx/blob/686ca2a8bb1fed8cd3a6dbe63d75923460f46888/src/onyx/peer/operation.clj#L27-L34

That applies not only to catalog functions, but lifecycle functions, trigger :sync functions, etc…

the old ns-resolve

as long as it can be resolved by that, you dont need to redeploy your jar

Yup 😄

so, the peers essentially just contain a bunch of functions

then you submit 'job' explaining them and how to wired them together

for a given purpose

Yup!

i think it got weird when i started seeing that the catalog was resubmitted each time

seemed like that info would just live on the peer

but, the parametrization is the clue

Slight tanget, is there a preferable way to submit a job to a cluster? Just contact one peer and submit-job ?

submit-job just talks to zookeeper, you dont need to run it on a peer.

Ahhh

It just needs the job datastructure

Of course

and uses the env-config to resolve zookeeper

@tcoupland: I’m not sure what your talking about, where were you seeing the catalog being resubmitted each time?

in the little starter project, the catalog is submitted with the workflow:

(let [job {:workflow looped-flow
               :catalog dev-catalog
               :lifecycles dev-lifecycles
               :flow-conditions flow-conditions
               :task-scheduler :onyx.task-scheduler/balanced}]
      (onyx.api/submit-job peer-config job)

@mccraigmccraig: yeah, it should work pretty much identically, so you can do what you did before

@tcoupland: Oh I see, that’s a common pattern among Onyx users. You have two entrypoints to your Uberjar, one that starts up the peer and one that submits the job. That makes it so you dont have to do anything extra during deployment except start your jar with a different entrypoint.

@gardnervickers: i think we're sorted now. Thanks, was a silly question really, there's no way it would have been made the other way, but sometimes you've just got to know

thanks for your help @gardnervickers

No problem! It’s totally not a silly question, Onyx has a different architecture than most are used too.

"so, the peers essentially just contain a bunch of functions .. then you submit 'job' explaining them and how to wired them together .. for a given purpose" - @tcoupland This is a great simple explanation.

@michaeldrogalis: thanks! feel free to use it