This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # admin-announcements (17)
- # alda (5)
- # aws (4)
- # beginners (19)
- # boot (80)
- # bristol-clojurians (3)
- # cider (15)
- # clojure (257)
- # clojure-berlin (1)
- # clojure-czech (1)
- # clojure-indonesia (1)
- # clojure-nl (1)
- # clojure-poland (1)
- # clojure-russia (77)
- # clojurescript (186)
- # core-async (7)
- # core-typed (6)
- # cursive (8)
- # datomic (2)
- # devcards (2)
- # editors (45)
- # emacs (23)
- # hoplon (83)
- # jobs (1)
- # ldnclj (106)
- # leiningen (43)
- # off-topic (4)
- # onyx (31)
- # reagent (10)
- # yada (1)
Riemann and datadog are very simple for integration configuration
And looks pretty cool
Thank you guys for you help and development
Integrating Riemann and datadog?
Together? Or you mean each are?
Hey, I am trying to write an output plugin against 0.7.5 of onyx. I was wondering how onyx finds the input or output functions in their respective namespaces for the different plugins. My assumption was that the catalog entry would have enough information to make that happen and for the seq plugin that appears to be the case but this is definitely not the case for the async plugin.
Hi chrisn, we're giving a workshop atm but I'll get back to you in a couple hours
Cool, basically you’re pointing to a builder function
when using the async input plugin do you need to put :done into the channel before closing it?
Specifically that set of lines makes is seem like you need to end on an even batch number with a single entry of :done.
You do if you want the job to complete by itself rather than having to kill it
@chrisn out of curiousity, what kind of output plugin are you building?
It calls a function generator at initialization based on a keyword. This allows me to specialize the function based on config variables (which we set via docker) and have it output whatever.
The generated function is called with a msg for each msg and then with nothing when the pipeline is shut down.
tomorrow we attempt to boot this onyx pipeline up on like 40 big 40 core ec2 instances to process a ton of data in under 2 hours (roughly).
Aha. Cool. So basically a Swiss Army knife output plugin like onyx-seq
Exactly. And the only thing I don't like about onyx-seq is that it always puts things into :elements instead of passing the seq values through directly.
Haha, sooooo we haven't tested Onyx with anything that big yet, so let us know what you find out. I hope you don't hit any pathological perf issues!
Ahhh. My suggestion would be to unwrap it with an onyx/fn on the input task. We do it that way just in case your seq is full of non-segments.
So, your test will run on 1600 cores? Whoa
How many gigabytes of images is that? Let us know if (when) you hit any issues with it - we haven't tested with anything that big yet. We're happy to help, though we're both at StrangeLoop atm
Its like 20 GB. It should work fine; I am testing locally and I am not hitting any problems. I don't pass images through Onyx just small maps describing things like one would expect.