I’m considering onyx for aggregating time-series data. Is performance comparable to Samza or Spark?

@joshg: The 0.8.0 release supporting aggregation first class will be out in 2-3 weeks. Performance, in our particular benchmark, is below what Samza offers, but not terribly far off - acceptable for most use cases. You should benchmark your particular workload and determine if it's acceptable for you.

@michaeldrogalis: that’s great to hear—onyx has a much more pleasant interface and I would like to avoid YARN if possible

@joshg: Understandable.

Gonna get some sleep. Happy Friday, or Saturday if you are east of me. 😛

(everyone’s east of you. everyone’s west of you, too 😁)

@spangler: OK, the main issue, i.e. the outer job completing before it should, is fixed. I pushed a snapshot of 0.8.0 up with the fix, we’ll backport it to 0.7 soon

Here’s my git diff for a working onyx-test

clojure/onyx-test - [master●●] » git diff diff --git a/project.clj b/project.clj index e95f228..e26e3ee 100644 --- a/project.clj +++ b/project.clj @@ -4,5 +4,5 @@ :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.7.0"] - [org.onyxplatform/onyx "0.7.11"]] + [org.onyxplatform/onyx "0.8.0-SNAPSHOT"]] :main onyx-test.core) diff --git a/src/onyx_test/core.clj b/src/onyx_test/core.clj index cf168c1..b69f87d 100644 --- a/src/onyx_test/core.clj +++ b/src/onyx_test/core.clj @@ -92,7 +92,7 @@ :onyx/batch-size batch-size :onyx/plugin :onyx.plugin.core-async/input :onyx/medium :core.async - :onyx/max-peers 4 + :onyx/max-peers 1 :onyx/pending-timeout 60000} {:onyx/name :a :onyx/type :function @@ -129,7 +129,7 @@ (def env-config {:zookeeper/address "127.0.0.1:2181" - :zookeeper/server? false + :zookeeper/server? true :zookeeper.server/port 2181 :onyx/id onyx-id}) @@ -138,11 +138,11 @@ :onyx/id onyx-id :onyx.peer/job-scheduler :onyx.job-scheduler/balanced :onyx.messaging/impl :aeron - :onyx.messaging.aeron/embedded-driver? false + :onyx.messaging.aeron/embedded-driver? true :onyx.messaging/peer-port-range [40600 40800] :onyx.messaging/bind-addr "localhost"}) -(def n-peers 11) +(def n-peers 12) (defn start-onyx []

@spangler: there may be another Onyx bug lurking in there, will look into everything further v soon

@spangler: I had a think about the jobs submitting jobs implementation, and I think it may be possible to do it well, but it’ll probably require some of the triggering/state management features coming in 0.8.0

Released Onyx 0.7.14 with @lucasbradstreet's fix.