Fork me on GitHub
#onyx
<
2016-03-30
>
sirsean20:03:18

Hi all. I’m trying to build a workflow that branches (such that at the end there are two output plugins). From everything I’ve read that should be totally possible. But any time I put a branch into the workflow, my jobs stop such that neither output is eventually hit.

sirsean20:03:50

For example, I took this:

[[:read-lines :original-wrapper]
   [:original-wrapper :determine-origin]
   [:determine-origin :build-row]
   [:build-row :prepare-rows]
   [:prepare-rows :write-lines]]
and turned it into this:
[[:read-lines :original-wrapper]
   [:original-wrapper :determine-origin]
   [:determine-origin :build-row]
   [:build-row :prepare-rows]
   [:build-row :prepare-rows-2]
   [:prepare-rows-2 :write-lines]
   [:prepare-rows :write-lines]]

sirsean20:03:30

The first workflow works as expected (the inputs are written to the DB). The second (in which :prepare-rows-2 is a copy in the catalog of :prepare-rows) workflow writes nothing to the DB.

sirsean20:03:20

Am I misunderstanding how branches work, and/or can :onyx/type :function not be a branch?

michaeldrogalis21:03:52

@sirsean: Yep, totally possible and standard usage. Are exceptions in onyx.log?

michaeldrogalis21:03:07

Oh - sorry, I missed your text under the code block. Reading..

michaeldrogalis21:03:42

Yep, your understanding is correct

sirsean21:03:48

I’m running in Docker. So there’s nothing in onyx.log on my host.

michaeldrogalis21:03:29

Okay. You're definitely going to want to figure out the log redirection to see what its says. If it grinds to a halt, you almost definitely have a stracktrace to look at.

sirsean21:03:31

The last line in the peer container’s log is

Started peers. Blocking forever.

michaeldrogalis21:03:55

@sirsean: Ah, that's telling. Notice that after you submit your job, the peers dont say that they're starting any work.

michaeldrogalis21:03:05

You probably need to start more peers. At least 1 per task, generally

michaeldrogalis21:03:33

e.g. your job never started in the first place before it didnt have enough resources.

sirsean21:03:06

Oh haha I have # peers at 6, so when I went from 5 to 7 tasks it didn’t work any more?

sirsean21:03:06

I’ll try that. (My original workflow with a second output had 6 tasks, this was just an attempt at a simplified version that ended up with 7. Do I need strictly more peers than tasks, or should it work if they’re equal?)

michaeldrogalis21:03:30

I know it seems dumb that you need to care about how many peers you launch, but it pays off down the road when you need to reason about the performance profile of your program in production.

sirsean21:03:23

I suppose that makes sense, you wouldn’t be able to tell what’s going on if one of your peers had to perform multiple tasks and the others didn’t.

michaeldrogalis21:03:30

@sirsean: Precisely. One peer == 1 task is an iron-clad guarantee.

sirsean21:03:12

For this case, that was the trick. 😄 (Now to go see what’s up with my more complicated plugin, but I’ll make sure N_PEERS is higher.)

sirsean21:03:15

Thank you!