Fork me on GitHub
#onyx
<
2016-09-01
>
mccraigmccraig11:09:05

@camechis i've done plenty with using https://github.com/elastic/elasticsearch-hadoop to load ES from cascalog/cascading jobs - worked fine without ES falling over. they weren't giant jobs, but not teeny either - generally with between 10^6 and 10^8 documents, and an ES cluster with 3 data nodes iirc

Travis12:09:25

@mccraigmccraig Cool, we are going to give it a shot

asolovyov12:09:59

I have a stream of indicated changes in entities coming through a Kafka topic. Onyx app fills them with more data and publishes to another Kafka topic. What's necessary is that changes for a single entity appear in order in a topic2 (relatively to one another). With a few peers allocated to a job order that order can be messed up. Do I understand correctly that grouping by function is the right way to deal with the situation?

lucasbradstreet12:09:34

@asolovyov: keeping things in order is tricky, especially with the current messaging model

lucasbradstreet12:09:43

The best you can currently do is using grouping, and windowing/state management to build up things in order and flush as you have everything you need

asolovyov12:09:16

the thing is that those entities are client orders

asolovyov12:09:23

so changes usually are pretty rare

asolovyov12:09:39

like maybe 1-2 a day while it's active

asolovyov12:09:49

but sometimes client orders and then almost immediately cancels

asolovyov12:09:01

or something like that

lucasbradstreet12:09:32

Do you need the whole history, or is it a kinda thing where one operation will supersede another? I imagine you need the whole history

zirmite12:09:30

What do you all think about the Apache Beam project? http://beam.incubator.apache.org/

zirmite12:09:21

It seems like an unnecessary layer on top of Onyx for clojurians but could be useful to hook into for adoption of Onyx, perhaps?

gardnervickers12:09:28

Onyx supports the underlying primitives of BEAM, our state model is based on the Dataflow paper which is kinda what spawned BEAM

zirmite13:09:16

right, onyx is very well aligned for that reason so that’s why I feel like BEAM is sort of unnecessary if you’re using Onyx as the “pipeline runner"

zirmite13:09:49

that said, being able to use BEAM to generate Onyx task bundles / jobs might be interesting

asolovyov13:09:59

@lucasbradstreet: unfortunately whole history

michaeldrogalis15:09:29

@zirmite Yes, I agree. I would like it very much if someone implemented the BEAM interface itself over Onyx.

michaeldrogalis15:09:47

If anyone's interested in tackling that, we'd be happy to help where we can.

zirmite15:09:36

@michaeldrogalis I’d be happy to contribute to that effort. would you spec out what that implementation should look like?

michaeldrogalis15:09:16

@zirmite I actually haven't had the time to dig into the specifics of the BEAM interface -- do you have a reference to it?

zirmite15:09:42

i only started looking at it today, but this doc seems like a good starting point: http://beam.incubator.apache.org/learn/programming-guide/

zirmite15:09:11

what I was thinking is a translation layer between the BEAM and Onyx primitives that describe the dag/workflows

michaeldrogalis15:09:24

Yeah, I think it was be an astonishingly small shim. All you need to do is spin up some Clojure data structures underneath the interface, and we already support almost all of their primitives 1-1

michaeldrogalis15:09:16

I've attempted something similar 3 or 4 times in experiments -- it ends up looking like a compiler, where each pass adds another task to the Onyx job.

aaelony22:09:54

question about the zooming patch... (https://github.com/onyx-platform/onyx-dashboard/commit/25d6692f3387608ac2b34a0077602bab0710996a) . I edited the project.clj to use [org.onyxplatform/onyx-visualization "0.2.0"] and restarted the dashboard. Is there a key combination to zoom? (maybe it's not that simple?) ?

michaeldrogalis23:09:19

@aaelony Not sure, I haven't tried it myself. Mouse wheel? Arrow keys?

aaelony23:09:37

yeah, I couldn't summon the powers of zoom. I need patience 🙂

aaelony23:09:30

I'll start it up again and try again...

aaelony23:09:17

yeah, neither mouse wheel, arrow keys, nor add to iTunes as a spoken track work for me