Onyx 0.7.11 has been released with an AOT bug fix.

@michaeldrogalis Okay! Everything is up and running, and I am launching onyx tasks successfully

I have a question about how to set up this process

in terms of decomposing what I have now into tasks

There is a kind of master process that gathers nodes for my graph

and then those nodes go through a variety of steps, each of which is independent and can be done in parallel

(a big reason I am using onyx)

@spangler: Okay.

I send them through in batches of about 30 nodes at a time

So the question is, do I want to make each of these separate jobs? Or can they be different steps in the same workflow?

What do you mean when you say nodes?

They are nodes in the graph

In the workflow?

the graph is the main data structure I am creating here

Ah no, just some data

I know the workflow is also an ADG

The data the workflow is processing is a graph composed of many nodes

Maybe ~1000-10000

There are some operations I need to do on the nodes themselves

then once all the nodes are processed, some operations to do on the graph as a whole

So, there is a process that progressively generates nodes, sending them 30 at a time to be further processed

Those can happen in parallel

Then once all that is done, a final processing of the whole graph

So you're saying your segments are themselves DAGs?

The segments for the most part are going to be nodes in a DG

Yes

Okay

Just wondering how to structure this in terms of onyx workflows

If this can all be one workflow?

Or if I need a separate job for processing the nodes

I didn't follow it closely enough to be able to give advice there. I'd need to see something more concrete.

Try one workflow, and see if there's anything preventing you from doing that.

Okay, I'll try that and probably be back with more specific questions : )

Thank you!

Np

Okay, hopefully this is more specific for you

I have a process that is generating 30 nodes at a time

Is there a way to send those 30 nodes onto the next stage in the workflow without ending the stage that is sending them?

That is my hangup right now

I know how to send a sequence of segments to a job, but not to a task from another task

Basically I want something like an core.async channel that I can put segments onto from one task and have them be input to the next task in the workflow

You dont control the flow of segments between tasks, Onyx makes that invisible to you.

You can use an output task to dump segments onto external storage, and have an input task from another job that reads that external storage. @spangler

Okay, so I do need another job then

That is the conclusion I've been arriving at

So it is okay to start another job from inside a running onyx job then?

It's a little unconventional, but there's nothing stopping you from doing that.

Okay

Probably easier to start both jobs at the same time, no?

That might end up being what I do

Next question:

I have an input that flows to multiple tasks

and then those tasks funnel to a single output

When I am sending segments, the last one is :done

Yup

So will take-segments! know when all of the tasks are done?

even though the :done is flowing to each task

Yes

Just trying to get everything straight here

Perfect!

You will not see :done at any stage in your tasks though. It will only appear at the end when everything is finished.

So no need to handle it in your functions

Right

It is just at that point take-segments! will return I am guessing?

Yeah

Okay, this might all work out

thank you for your help as I orient myself

Sure thing.

Okay, things are working! (kind of)

A couple issues:

1. Onyx seems to attempt to start my task again before it is done, causing problems

2. If I run the job once it is fine, but if I have two of the same job running at the same time I run out of virtual peers

In that I get this message over and over again

Not enough virtual peers have warmed up to start the task yet, backing off and trying again...

This is with running about 50 virtual peers, which seems like a lot since my workflows only have a total of 9 steps in them

If I try to add more virtual peers I get this error on startup

org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

over and over again, after ~50 of them succeed

Then I start getting a lot of these errors

15:44:30.500 [clojure-agent-send-pool-3-SendThread(localhost:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_67]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_67]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_67]
	at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.7.0_67]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_67]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66) ~[zookeeper-3.4.1.jar:3.4.1-1212694]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291) ~[zookeeper-3.4.1.jar:3.4.1-1212694]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041) ~[zookeeper-3.4.1.jar:3.4.1-1212694]

So I think zookeeper has some kind of limit that kicks in around 50-60 virtual peers?

@michaeldrogalis

Hmm.... I upgraded to the latest onyx (0.7.11)

and now on startup the process just hangs here

16:09:57.332 [clojure-agent-send-pool-3] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@154cfab2
16:09:57.354 [clojure-agent-send-pool-3-SendThread(127.0.0.1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16:09:57.362 [clojure-agent-send-pool-3-SendThread(127.0.0.1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
16:09:57.370 [clojure-agent-send-pool-3-SendThread(127.0.0.1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x150675a060e085d, negotiated timeout = 40000
16:09:57.377 [clojure-agent-send-pool-3-EventThread] INFO  o.a.c.f.state.ConnectionStateManager - State change: CONNECTED

Hasn't moved for almost 10 minutes