This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-09-18
Channels
- # aleph (45)
- # aws (4)
- # beginners (56)
- # boot (2)
- # cider (45)
- # clara (2)
- # cljs-dev (9)
- # cljsrn (31)
- # clojure (71)
- # clojure-dusseldorf (8)
- # clojure-gamedev (1)
- # clojure-italy (22)
- # clojure-nl (1)
- # clojure-russia (46)
- # clojure-sg (1)
- # clojure-spec (5)
- # clojure-uk (40)
- # clojurescript (30)
- # community-development (3)
- # cursive (17)
- # data-science (1)
- # datomic (18)
- # emacs (3)
- # figwheel (1)
- # fulcro (19)
- # hoplon (12)
- # jobs (5)
- # leiningen (42)
- # off-topic (12)
- # om (2)
- # onyx (41)
- # re-frame (19)
- # ring-swagger (1)
- # rum (3)
- # shadow-cljs (4)
- # specter (7)
- # unrepl (2)
- # vim (25)
- # yada (24)
I’d stick with a minimum number of topics as possible and filter from there @camechis - MapR iterate over and over in a sales manner about Kafka’s performance drop with 1000's of topics. 100 would be fine just hard to manage from a Onyx point of view.
does onyx supports microbatching and stream checkpoints ? I assume i can do it manually in my workflow, but I want to know " does onyx provides such stuff out of the box" ?
@lxsameer Onyx uses a different algorithm from Spark-style microbatching but it supports the same kind of workloads, allowing exactly-once message processing and stateful aggregations.
@lxsameer The aggregation and state management docs may help http://www.onyxplatform.org/docs/user-guide/0.11.x/#aggregation-state-management
If you’re interested in the algorithm behind stateful aggregations this blog post goes into it a bit http://www.onyxplatform.org/jekyll/update/2017/07/10/Onyx-Asynchronous-Barrier-Snapshotting.html
awesome thanks @gardnervickers
Onyx is getting first-class complex event processing integration. Took our first step today and open sourced the backing library. https://github.com/pyroclastio/metamorphic
We’ve actually got it working at scale with Onyx already, albeit undocumented. https://github.com/onyx-platform/onyx-cep
onyx-cep is 404 / private
Feature looks really nice.
Eeek - one sec.
Sorry about that. Fixed @eriktjacobsen
interesting. does this essentially compile event processing "patterns" into a highly optimised onyx workflow ?
@lmergen Tacks right onto existing Onyx jobs https://github.com/onyx-platform/onyx-cep/blob/master/test/onyx/cep/smoke_test.clj#L93
Only used for matching.
cool. i can see some use cases here. how well does this scale when it needs to match over large quantities of data ?
Metamorphic is single-process. It works as Onyx normally does when used with onyx-cep
Just need an appropriate way to shard your data
It does need more performance testing for large and complex matches. There’s likely some low hanging fruit there
It’s designed for matching over unbounded streams, so algorithmically it’s a good fit.
Sweet.
We have the disk backed state store now, and we can accumulate possible matches and then flush when watermarks come in
It’s not in a great state for it, but we have escape hatches.
Oh yeah 🙂
It’s alpha quality, but yes.
We need to hammer on it in prod.
Is onyx-spec ready for consumption yet? Any examples on how I would use it to validate job definitions?
@dave.dixon The specs in onyx-spec have been around for a long time. We only recently moved them out
We havent done much with them because we didnt want a hard dependency on Clojure 1.9 while it’s in alpha.
The specs haven’t changed for some time, and we use them in other projects that depend on Onyx. We’re generally validating against particular specs rather than the entire job in one shot.
Ok, I'll give them a try.