watching it, feeling amazing 😄

Great talk @michaeldrogalis

michaeldrogalis: very curious to see how things are going to move in a flink like direction

Is there somewhere an example or test that demonstrates message retrying in case it is not fully acked?

@aspra: we really need a better way to test for this. The best example I’ve got uses a lifecycle to crash the peer and restart, then the message will retry. This example is in the onyx-datomic plugin https://github.com/onyx-platform/onyx-datomic/blob/0.9.x/test/onyx/plugin/input_fault_tolerance_test.clj

Note in this example we use lifecycle/handle-exception to tell the peer to restart and continue processing the job, rather than killing the job

https://github.com/onyx-platform/onyx-datomic/blob/0.9.x/test/onyx/plugin/input_fault_tolerance_test.clj#L30

Right. I was more thinking of no peer or job restarting but more for example an error response to an http request

I f I understand correctly then it has to do with the retry-segment implementation of the input plugin used?

@nxqd @bcambel Thanks!!

@otfrom: Should be a pretty transparent change for users. One big win beyond a performance boost will be fully automatic backpressure. Very little tuning to do.

@aspra: in that case it should automatically occur after the pending-timeout is hit on the input task

is pending-timeout the same as batch-timeout?

They’re different. http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-timeout vs http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/pending-timeout

thinking about adding peers to a cluster; lets say I package my-code-0.1.0 in a JAR, throw it on a node in the cluster, have it boot up and connect to ZK and then repeat the process but with my-code-0.1.1 - other than this being totally avoidable with good ops, is there anything Onyx does to warn/prevent this?

@acron: There's nothing preventing you from doing that. I'd call that a rolling upgrade tbh 😛

heh, ok, thanks. Great talk @ Clojure/West btw

Thanks

Next release of Onyx is getting an upgraded static analyzer to detect job-level errors before you submit them. Here's a little preview..

@michaeldrogalis: I know at one point you were looking for help on a “scheduler” At the time i didn’t really understand what this meant. I have come to understand this refers to something analogous to what YARN or MESOS does. Is that correct? Did you finally settle on a solution? I looked around for material on this and came up short. Thanks!

@drewverlee: Onyx runs compatibly with Mesos and other tools that offer a "Level 1 scheduler", e.g. machine level allocation across different contending resources. We have our own "Level 2 scheduler", which has application specific knowledge about tasks and such - things that Mesos-ish couldn't anticipate. We went with a library called BtrPlace.

Crazy strong library, highly recommended. http://www.btrplace.org/