Hi guys 🙂

So, question, does onyx use ZK as consistent DB only?

And for consensus about when peers go offline, via watches on ephemeral nodes

How much data Onyx stores into DB and what kind?:)

It writes log messages for peer coordination, and data chunks which contain the job data containing the data that defines the job. This is of varying size and depends on what your job looks like. If this gets too big it's possible to pull it out of the job definition and put it on s3 to be pulled down when tasks start up

Now I'm looking for how to increase performance of ZK cluster

Found interesting article http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zookeeper_psuedo_scalability_and_absolute

Other words cluster is more for fault tolerance rather than for performance

Observers are supposed to be one solution http://blog.cloudera.com/blog/2009/12/observers-making-zookeeper-scale-even-further/

thank you, looking

lucasbradstreet: I suppose the case are better for a queue?

I'm not sure ZK is good for this

i mean the case of data chunks

just though

We’ll likely implement a second chunk backend that would allow you to read and write that data to S3 instead

Excellent!

lucasbradstreet: you rock, much appreciate it

http://s2.quickmeme.com/img/22/223c6f679469cdce07f73c599aa459d7da6520054b390621af09e91b8716d156.jpg

😄

@lucasbradstreet: Is there anything special that needs to be done in order to make the Dashboard display Job information and other related info? Right now the only thing I see in the dashboard are the peer events and when a Job gets submitted but nothing else. We are attempting to debug a job and its very tricky when its spread across peers and I am not seeing any errors, lol.

after more digging around, i can see that tweets are coming through by adding logging on the tweet callback handler at https://github.com/onyx-platform/onyx-twitter/blob/master/src/onyx/plugin/twitter.clj#L62. but it seems like next-state is never run at https://github.com/onyx-platform/onyx-twitter/blob/master/src/onyx/plugin/twitter.clj#L76. i've added logging there as well but it never seems to run. update: i left this running and the process ran out of memory which seems to further indicate that data is coming in but not being processed further

Datomic log API supports in mem now. Nice! http://blog.datomic.com/2016/08/log-api-for-memory-databases.html?m=1. At some point we can remove that restriction from onyx-datomic

Any ideas on what this means

WARN [onyx.messaging.aeron] - 
                                  java.lang.Thread.run           Thread.java: 745
    uk.co.real_logic.agrona.concurrent.AgentRunner.run      AgentRunner.java: 105
         uk.co.real_logic.aeron.ClientConductor.doWork  ClientConductor.java: 113
         uk.co.real_logic.aeron.ClientConductor.doWork  ClientConductor.java: 293
uk.co.real_logic.aeron.ClientConductor.onCheckTimeouts  ClientConductor.java: 338
uk.co.real_logic.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns

16-Aug-10 20:04:44  WARN [onyx.messaging.aeron.publication-manager] - Aeron messaging publication error: uk.co.real_logic.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
16-Aug-10 20:04:44  WARN [onyx.messaging.aeron.publication-manager] - Aeron messaging publication error: uk.co.real_logic.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
16-Aug-10 20:04:44  WARN [onyx.messaging.aeron.publication-manager] - Aeron messaging publication error: uk.co.real_logic.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
16-Aug-10 20:04:44  WARN [onyx.messaging.aeron] - 

this is spewing over and over on one of my peers

after several hours

Generally this happens only when you start getting memory pressure issues, and are spending a lot of your time garbage collecting

You can increased the timeout, which can help things, but you probably need to solve your underlying issue.

Gotta sleep

no worries