Fork me on GitHub
#onyx
<
2015-10-19
>
spangler19:10:49

@michaeldrogalis Okay another onyx question: I have a (stuart sierra style) component with a bunch of state in it, things like the db connection and a kafka channel, etc... What is the best way to access that from an onyx job?

spangler19:10:02

I see there are parameters you can pass in as part of the catalog

spangler19:10:37

but then I remember you saying all onyx workflows, catalogs and lifecycles are serializable things

spangler19:10:27

So I probably can't put a component in there

spangler19:10:06

Where should I provide access to the component so that I can use it in my onyx tasks?

michaeldrogalis20:10:13

@spangler: Correct, Onyx job data structures need to be serializable. You'd construct the Component through a lifecycle.

michaeldrogalis20:10:54

Note that you need to build the Component (e.g. call Start on it) within the lifecycle. Onyx intentionally makes it very hard to reach into a global state and gain access to an already running component.

spangler20:10:36

@michaeldrogalis Ah okay, so every time a task fires up it will build the component and then tear it down when it is done?

spangler20:10:44

Okay cool, I can do that. Another quick question: if my job branches into many tasks in the workflow, is each task potentially distributed among the virtual peers?

spangler20:10:05

I am assuming this is so, just checking

michaeldrogalis20:10:14

Can you rephrase that?

michaeldrogalis20:10:31

So like, 100 virtual peers, 10 tasks, ~ 10 vpeers per task?

spangler20:10:12

Ah, I was just wondering if within the same call to submit job, if the various tasks in the workflow are distributed out to all the available virtual peers?

spangler20:10:34

Or potentially at least

michaeldrogalis20:10:02

That's how it works, yes. You can control it via schedulers if you need anything other than a balanced workload: https://github.com/onyx-platform/onyx/blob/master/doc/user-guide/scheduling.md

spangler20:10:31

Okay, that is what I assumed, just wanted to make sure I was thinking about it correctly