Fork me on GitHub
#onyx
<
2017-05-25
>
stephenmhopper15:05:09

I’m rewriting a Clojure data processing pipeline that uses core async channels and Amazon SQS. There’s a lot of extra, error prone, hand-written logic for handling things like errors and whatnot that Onyx handles out of the box. I’d like to just move this thing straight over to Onyx, but I have a couple questions. To start with, is it possible to just run Onyx in memory without external ZK or bookkeeper instances? I know that onyx-local-rt exists, but I seem to recall a situation wherein in didn’t work with external message sources (like SQS). What’s the best way to do this?

stephenmhopper15:05:50

Eventually, we’ll scale this thing out to do distributed processing across multiple machines but for now, the workload is small enough that one machine should be able to do the trick for the initial rewrite

Travis16:05:41

You can run onyx locally using the testing framework. If your starting from scratch you don’t need bookeeper anymore with 0.10. ZK will start up on its own

lucasbradstreet16:05:58

no onyx-local-rt required.

michaeldrogalis16:05:03

@stephenmhopper If you do want to use local-rt, you can put core.async channels, SQS producers consumers, or whatever on either ends of the input and output tasks.

lucasbradstreet16:05:28

@michaeldrogalis is there any advantage in using local-rt though?

michaeldrogalis16:05:31

It’s just a data structure, so you can loop over it with api/tick and add messages from the outside when you receive ‘em.

lucasbradstreet16:05:42

seems like it’d just be easier to run regular onyx locally.

michaeldrogalis16:05:46

Nah not really, just answering that particular question.

lucasbradstreet16:05:57

Oki, sure thing.

stephenmhopper16:05:20

Cool, thanks everybody!

lucasbradstreet16:05:21

Figured if they’re going to scale it out later they might as well use real onyx on a single node.

lucasbradstreet16:05:53

local rt may be a good first step though.

stephenmhopper18:05:04

@camechis You mentioned that bookeeper is no longer required with 0.10. I was planning on using the config file in the onyx-template repo as a starting point. Can I just remove the bookeeper entries?

Travis18:05:19

My guess is yes, I am not sure what the status of the template is in relation to 0.10

lucasbradstreet18:05:36

@stephenmhopper yes. I should remove those myself

stephenmhopper18:05:15

Cool. Have any of the other config options changed since 0.9?

lucasbradstreet18:05:36

Done. They won’t be in the next release.

lucasbradstreet18:05:40

None in the config.edn

stephenmhopper18:05:13

Is there a “What’s new in 0.10” post / doc somewhere?

stephenmhopper18:05:39

oh, cool, I’ll use that

lucasbradstreet18:05:06

there were lots of config deprecations in 0.10, but most of them weren’t touched by users. It’s settled down now.

stephenmhopper18:05:45

Yeah, and it looks like most of the new config options are ABS related fine tunings?

stephenmhopper18:05:56

Changes to the core async plugin

:done messages are no longer supported on core async input channels. Simply close the channel instead of putting on a done.

In addition, :done messages are no longer written to output channels. Insead use onyx.api/await-job-completion or onyx.test-helper/feedback-exception! to wait for the job to end, and then drain the output channels until no more messages can be read.
^I like that

michaeldrogalis18:05:03

Haha, yeah sorry. The sentinel thing was a bad idea.

stephenmhopper18:05:48

I never ran into any real problems with it. It just required a little bit of extra thinking. Now I can just strip that logic out!