This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-09-01
Channels
- # admin-announcements (1)
- # aws (1)
- # beginners (14)
- # boot (19)
- # cljs-dev (10)
- # cljsrn (2)
- # clojure (64)
- # clojure-android (4)
- # clojure-dev (5)
- # clojure-greece (7)
- # clojure-italy (10)
- # clojure-russia (42)
- # clojure-spec (117)
- # clojure-uk (78)
- # clojurescript (160)
- # cloverage (1)
- # conf-proposals (1)
- # cursive (8)
- # datomic (93)
- # editors (8)
- # editors-rus (5)
- # figwheel (1)
- # flambo (14)
- # hoplon (95)
- # jobs (2)
- # jobs-rus (1)
- # lambdaisland (4)
- # lein-figwheel (6)
- # leiningen (3)
- # om (106)
- # onyx (33)
- # planck (6)
- # proton (3)
- # protorepl (2)
- # random (2)
- # re-frame (9)
- # reagent (5)
- # ring (1)
- # untangled (61)
- # yada (50)
@camechis i've done plenty with using https://github.com/elastic/elasticsearch-hadoop to load ES from cascalog/cascading jobs - worked fine without ES falling over. they weren't giant jobs, but not teeny either - generally with between 10^6 and 10^8 documents, and an ES cluster with 3 data nodes iirc
@mccraigmccraig Cool, we are going to give it a shot
I have a stream of indicated changes in entities coming through a Kafka topic. Onyx app fills them with more data and publishes to another Kafka topic. What's necessary is that changes for a single entity appear in order in a topic2 (relatively to one another). With a few peers allocated to a job order that order can be messed up. Do I understand correctly that grouping by function is the right way to deal with the situation?
@asolovyov: keeping things in order is tricky, especially with the current messaging model
The best you can currently do is using grouping, and windowing/state management to build up things in order and flush as you have everything you need
Do you need the whole history, or is it a kinda thing where one operation will supersede another? I imagine you need the whole history
What do you all think about the Apache Beam project? http://beam.incubator.apache.org/
It seems like an unnecessary layer on top of Onyx for clojurians but could be useful to hook into for adoption of Onyx, perhaps?
Onyx supports the underlying primitives of BEAM, our state model is based on the Dataflow paper which is kinda what spawned BEAM
right, onyx is very well aligned for that reason so that’s why I feel like BEAM is sort of unnecessary if you’re using Onyx as the “pipeline runner"
that said, being able to use BEAM to generate Onyx task bundles / jobs might be interesting
@lucasbradstreet: unfortunately whole history
@zirmite Yes, I agree. I would like it very much if someone implemented the BEAM interface itself over Onyx.
If anyone's interested in tackling that, we'd be happy to help where we can.
@michaeldrogalis I’d be happy to contribute to that effort. would you spec out what that implementation should look like?
@zirmite I actually haven't had the time to dig into the specifics of the BEAM interface -- do you have a reference to it?
i only started looking at it today, but this doc seems like a good starting point: http://beam.incubator.apache.org/learn/programming-guide/
what I was thinking is a translation layer between the BEAM and Onyx primitives that describe the dag/workflows
Yeah, I think it was be an astonishingly small shim. All you need to do is spin up some Clojure data structures underneath the interface, and we already support almost all of their primitives 1-1
I've attempted something similar 3 or 4 times in experiments -- it ends up looking like a compiler, where each pass adds another task to the Onyx job.
question about the zooming patch... (https://github.com/onyx-platform/onyx-dashboard/commit/25d6692f3387608ac2b34a0077602bab0710996a) . I edited the project.clj to use [org.onyxplatform/onyx-visualization "0.2.0"]
and restarted the dashboard. Is there a key combination to zoom? (maybe it's not that simple?) ?
@aaelony Not sure, I haven't tried it myself. Mouse wheel? Arrow keys?
yeah, neither mouse wheel, arrow keys, nor add to iTunes as a spoken track work for me
@mariusz_jachimowicz Might know =o