Fork me on GitHub

watching it, feeling amazing 😄


michaeldrogalis: very curious to see how things are going to move in a flink like direction


Is there somewhere an example or test that demonstrates message retrying in case it is not fully acked?


@aspra: we really need a better way to test for this. The best example I’ve got uses a lifecycle to crash the peer and restart, then the message will retry. This example is in the onyx-datomic plugin


Note in this example we use lifecycle/handle-exception to tell the peer to restart and continue processing the job, rather than killing the job


Right. I was more thinking of no peer or job restarting but more for example an error response to an http request


I f I understand correctly then it has to do with the retry-segment implementation of the input plugin used?


@otfrom: Should be a pretty transparent change for users. One big win beyond a performance boost will be fully automatic backpressure. Very little tuning to do.


@aspra: in that case it should automatically occur after the pending-timeout is hit on the input task


is pending-timeout the same as batch-timeout?


thinking about adding peers to a cluster; lets say I package my-code-0.1.0 in a JAR, throw it on a node in the cluster, have it boot up and connect to ZK and then repeat the process but with my-code-0.1.1 - other than this being totally avoidable with good ops, is there anything Onyx does to warn/prevent this?


@acron: There's nothing preventing you from doing that. I'd call that a rolling upgrade tbh 😛


heh, ok, thanks. Great talk @ Clojure/West btw


Next release of Onyx is getting an upgraded static analyzer to detect job-level errors before you submit them. Here's a little preview..

Drew Verlee17:04:38

@michaeldrogalis: I know at one point you were looking for help on a “scheduler” At the time i didn’t really understand what this meant. I have come to understand this refers to something analogous to what YARN or MESOS does. Is that correct? Did you finally settle on a solution? I looked around for material on this and came up short. Thanks!


@drewverlee: Onyx runs compatibly with Mesos and other tools that offer a "Level 1 scheduler", e.g. machine level allocation across different contending resources. We have our own "Level 2 scheduler", which has application specific knowledge about tasks and such - things that Mesos-ish couldn't anticipate. We went with a library called BtrPlace.


Crazy strong library, highly recommended.