Fork me on GitHub
#onyx
<
2016-09-27
>
aengelberg02:09:21

Looking through the onyx-kafka code, it looks like onyx-kafka completely ignores kafka offset management in favor of using onyx checkpoint management in zookeeper. Is this accurate?

lucasbradstreet05:09:43

Yes, that is right.

aspra08:09:52

when upgrading to 0.9.10 it seems that the test-helper cannot find the default "test-config.edn” anymore

lucasbradstreet08:09:46

Hi @aspra. You are using onyx.test-helper/load-config in your tests?

lucasbradstreet08:09:27

Hmm. I don’t think we’ve changed anything about how that works. Is it possible that test-config.edn was moved out of your resources directory?

aspra08:09:23

let me have a look. I also can’t find anything in your changes… so maybe something from here

aspra09:09:56

very strange. It works until I upgrade onyx but I cant see where it comes from

lucasbradstreet09:09:15

Definitely weird. Which version are you upgrading from?

lucasbradstreet09:09:09

Ah. I think I know what it is. I think we may have moved test-config.edn to test-resources and added it to :resource-paths ["test-resources/“] for dev

lucasbradstreet09:09:51

I’m not sure we really intended anyone to use load-config on their projects.

lucasbradstreet09:09:32

I think you would be best copying the end file somewhere into your resources and maintain it yourself, so you can have more control over the test setup

aspra09:09:01

yeah makes sense. Will do

lucasbradstreet09:09:19

you’re welcome

lucasbradstreet09:09:50

I’d name it something different and load it with the 1 arity function so they can’t conflict

lucasbradstreet10:09:04

@zamaterian thanks. Review is up. Mostly small stuff 🙂

lucasbradstreet10:09:10

Great, looks great. I’m just going to change the base branch to master, so some of the comments may be lost.

lucasbradstreet10:09:04

Cool. CI will release a snapshot for you to use until we release

lucasbradstreet10:09:11

Thanks. Great stuff 🙂

zamaterian16:09:16

@michaeldrogalis my little contribution stand on top of giants onyx 🙂

michaeldrogalis16:09:44

@zamaterian Indeed. Rich has strong shoulders. 😉

michaeldrogalis17:09:13

@mariusz_jachimowicz Same to you. Thanks for all the awesome pull requests.

michaeldrogalis21:09:32

Just put support into master for :onyx/batch-fn?. Per the docstring:

Boolean value indicating whether the function in this catalog entry denoted by `:onyx/fn` should take a single segment, or the entire batch of segments that were read as a parameter. When `true`, the `:onyx/fn` must return a sequence of the same length as its input match. Each element of the return value represents the children segments that will succeed the corresponding parent segment. Hence, the arguments match positionally. Children values may either be a single segment, or a vector of segments, as normal. This feature is useful for batching requests to services, waiting for whole batches of asynchronous requests to be made, dedepulicating calculations, etc.
Thanks to @smw and @lucasbradstreet. Snapshot being deployed from CI in a few minutes.

michaeldrogalis23:09:36

@lucasbradstreet Let's get a release out tomorrow. I updated the User Guide for batch-fn, and I also made a branch in learn-onyx to replace the bulk challenge with a new challenge for batch-fn.

michaeldrogalis23:09:56

Ill handle updating learn-onyx when we're ready since the merge logic is a little tricky when updating both test and src.