Fork me on GitHub
#onyx
<
2016-03-04
>
acron14:03:16

I'm working through the learn-onyx challenges and notice that if I mess one up, the tests hang - what's the cause of this? is this avoidable?

lucasbradstreet14:03:48

Usually it’ll be something that’s logged in onyx.log

lucasbradstreet14:03:54

Are you using the repl to run the tests?

lucasbradstreet14:03:48

Generally they hang because it’s waiting to read the segments back but they never arrive. If you’re using the repl and viewing the onyx.log you can generally ctrl-c, fix it, and rerun the test from the repl

acron14:03:33

Ooh, I wasn't looking at onyx.log

lucasbradstreet14:03:32

We have an extra test-helper function called feedback exception, that reads the serialized exception back from ZooKeeper, which we should start using in learn-onyx. For now you need the onyx.log simple_smile

acron14:03:48

Ahh cool ok, yes I see an exception in the log simple_smile

lucasbradstreet14:03:49

I recommend tail -F on the onyx.log while you dev

acron14:03:17

I'm using CIDER btw

acron14:03:35

Works well but a bit more blah required when the tests hang

acron14:03:48

But no that's super helpful, thanks

michaeldrogalis15:03:00

@acron: C-c C-b should unstuck you, unless Cider changed the bindings since the last time I used it

acron15:03:16

@michaeldrogalis: wow, I never knew that! ever learning in the land of Clojure.

acron15:03:21

thanks again

gardnervickers19:03:31

@lucasbradstreet: For the task bundles regarding the schemas, could you refresh my memory? We’re no longer merging with os/TaskMap correct? https://gist.github.com/gardnervickers/7705757653df28dd0e49

gardnervickers19:03:47

or os/Lifecycle

lucasbradstreet19:03:45

It’s a conditional that allows us to use different schema definitions for different types of tasks e.g. input/output

gardnervickers19:03:34

Hmm I wonder why I didn’t notice that before.

lucasbradstreet19:03:52

Note the UserTaskMap key. That basically says, allow any other key than a :datomic one

lucasbradstreet19:03:08

That way they can’t make :datomic/whatever typos and it’ll be allowed

gardnervickers19:03:43

I did not know you could do stuff like that with Schema. I’ll take a look at how it’s implemented.

lucasbradstreet19:03:23

Yeah, the conditional part is important for things like allowing :onyx/group-by-key XOR :onyx/group-by-fn

lucasbradstreet19:03:15

I prefer to default a lot of those settings, e.g. fetch size

lucasbradstreet19:03:49

Would rather just stick them in the base task map - they can be overridden by the opts if needed

lucasbradstreet19:03:27

For Kafka I think at least these should just be the defaults in the readme:

lucasbradstreet19:03:32

:kafka/chan-capacity chan-capacity :kafka/fetch-size fetch-size :kafka/empty-read-back-off empty-read-back-off :kafka/commit-interval commit-interval

gardnervickers19:03:14

I thought we were putting the option in the function signature no?

gardnervickers19:03:31

so that IDE’s could pick them up when using the function.

lucasbradstreet19:03:35

I was thinking the mandatory options should be

gardnervickers19:03:45

Yea I like that better

lucasbradstreet19:03:49

Otherwise you’re creating more boilerplate

gardnervickers19:03:35

I remember we decided something else around the signature’s too, we would have two arity’s, one where the required args were seperate arguments and another where they were :keys in the opts map. Do we still want to do it like that?

lucasbradstreet19:03:10

It’s a little more work but I’m happy to do that

lucasbradstreet19:03:16

There’s not much downside

lucasbradstreet19:03:33

I have come around to almost never wanting the non-task-opts version but I can see why people might want it

gardnervickers19:03:57

By non-task-opts version you mean the combined map version?

lucasbradstreet19:03:59

I think if you go that route, you should have the mandatory args in one signature

lucasbradstreet19:03:16

And that’s kind of the documentation for what should be in the second version

lucasbradstreet19:03:20

I’ll point you to what I mean

gardnervickers19:03:38

Thanks, I know we talked about this before but for some reason I’m blanking super hard

lucasbradstreet19:03:57

No problem. We gotta get it right simple_smile

lucasbradstreet19:03:53

You take the mandatory args in one signature

gardnervickers19:03:54

Ah gotcha now I remember

gardnervickers19:03:49

Probably should make a macro for this eventually

gardnervickers19:03:20

Since it’s pretty much the same pattern for every one of them

lucasbradstreet19:03:52

Do macros give you much though?

lucasbradstreet19:03:07

I don’t see much that couldn’t be done with a function

gardnervickers19:03:39

Yea you’d end up writing just as much