Fork me on GitHub
#onyx
<
2015-09-24
>
Kira Sotnikov14:09:41

Riemann and datadog are very simple for integration configuration

Kira Sotnikov14:09:45

And looks pretty cool

Kira Sotnikov14:09:27

Thank you guys for you help and development

lucasbradstreet15:09:17

Integrating Riemann and datadog?

lucasbradstreet15:09:46

Together? Or you mean each are?

chrisn17:09:31

Hey, I am trying to write an output plugin against 0.7.5 of onyx. I was wondering how onyx finds the input or output functions in their respective namespaces for the different plugins. My assumption was that the catalog entry would have enough information to make that happen and for the seq plugin that appears to be the case but this is definitely not the case for the async plugin.

lucasbradstreet17:09:36

Hi chrisn, we're giving a workshop atm but I'll get back to you in a couple hours

chrisn18:09:08

Cool, np. I think I have it.

lucasbradstreet18:09:31

Cool, basically you’re pointing to a builder function simple_smile

chrisn21:09:30

when using the async input plugin do you need to put :done into the channel before closing it?

chrisn21:09:12

Specifically that set of lines makes is seem like you need to end on an even batch number with a single entry of :done.

lucasbradstreet21:09:14

You do if you want the job to complete by itself rather than having to kill it

chrisn21:09:45

lol k. Yes, that works.

lucasbradstreet22:09:50

@chrisn out of curiousity, what kind of output plugin are you building?

chrisn22:09:15

Ideally the last one I ever write.

chrisn22:09:50

It calls a function generator at initialization based on a keyword. This allows me to specialize the function based on config variables (which we set via docker) and have it output whatever.

chrisn23:09:34

The generated function is called with a msg for each msg and then with nothing when the pipeline is shut down.

chrisn23:09:56

It is currently working simple_smile.

chrisn23:09:55

tomorrow we attempt to boot this onyx pipeline up on like 40 big 40 core ec2 instances to process a ton of data in under 2 hours (roughly).

lucasbradstreet23:09:09

Aha. Cool. So basically a Swiss Army knife output plugin like onyx-seq

chrisn23:09:41

Exactly. And the only thing I don't like about onyx-seq is that it always puts things into :elements instead of passing the seq values through directly.

lucasbradstreet23:09:57

Haha, sooooo we haven't tested Onyx with anything that big yet, so let us know what you find out. I hope you don't hit any pathological perf issues!

chrisn23:09:02

This makes it a less than ideal drop in replacement for e.g. the async plugin

lucasbradstreet23:09:42

Ahhh. My suggestion would be to unwrap it with an onyx/fn on the input task. We do it that way just in case your seq is full of non-segments.

lucasbradstreet23:09:00

So, your test will run on 1600 cores? Whoa

chrisn23:09:07

Ideally. Image processing.

chrisn23:09:14

800,000 images.

lucasbradstreet23:09:18

How many gigabytes of images is that? Let us know if (when) you hit any issues with it - we haven't tested with anything that big yet. We're happy to help, though we're both at StrangeLoop atm

chrisn23:09:53

Its like 20 GB. It should work fine; I am testing locally and I am not hitting any problems. I don't pass images through Onyx just small maps describing things like one would expect.

chrisn23:09:16

OK, well thanks for the offer I will let you know if I hit something odd.