Fork me on GitHub
#onyx
<
2017-04-27
>
jetmind10:04:52

@michaeldrogalis Hey! I have created a repo with minimal reproduction of the problem with lazy seqs we discussed yesterday. Here’s the issue https://github.com/onyx-platform/onyx/issues/755

georgek13:04:38

This is probably a simple question: I’m hitting a ‘unable to resolve symbol’ when trying to run a workflow in test. Right now the job is spun up in test. It contains a catalog entry that refer’s to a function from a namespace in the main codebase. Since that function is only represented as a catalog entry I don’t refer/import it anywhere directly in the test namespace. When the job runs that fn can’t be found. How do I ensure that it and others like it can be resolved?

georgek13:04:04

So, it resolves it if I explicitly require the function in the enclosing namespace that runs the job. Is there any other way to ensure it can be found?

michaeldrogalis15:04:45

@jetmind Thanks! I’ll get a look at it tonight.

michaeldrogalis15:04:16

@georgek That’s about it. It’s the same way as in Clojure. To ensure something is resolvable on the classpath, you need to require it.

georgek15:04:48

Kewl, that’s what I suspected. 🙂

kjothen16:04:44

I have hit a problem with punctuation triggers working on 0.10.beta9. From what I can tell, onyx.peer.task-lifecycle/checkpoint-state throws an "Unfreezable type: class clojure.lang.Var" exception, possibly because it's trying to serialize the trigger's :pred-fn? Exception is thrown with any predicate function I use, even (constantly false). Not entirely sure what to do next, other than remove the trigger entirely and workaround it using another trigger strategy.

michaeldrogalis16:04:35

@kjothen Can you upgrade to beta12 and try to reproduce it?

michaeldrogalis16:04:39

@kjothen I’ll take a look, that’s a perplexing one. We don’t serialize the pred-fn in the checkpoints

michaeldrogalis16:04:24

If you can Gist us the full stack trace, that’d be appreciated. Also your window & trigger definition.

kjothen16:04:28

@michaeldrogalis Same behaviour under beta12 too. The :pred-fn was just a (bad) guess, I'll prepare the gist (which is tricky because my work PC cannot access slack or Gist!) If it helps, I have a global window using the built in conj aggregator and the (new!) emit trigger work. I'm only using the punctuation trigger because I need to ensure the window state is emitted downstream before the job competes.

michaeldrogalis16:04:42

@kjothen If the stack trace blew up at onyx.peer.task-lifecycle/checkpoint-state, you probably have something in your Conj aggregate that can’t be deserialized.

michaeldrogalis16:04:54

I’m not sure how Nippy was able to freeze something it couldn’t thaw, though.

michaeldrogalis16:04:13

What types of values are you putting into the window?

lucasbradstreet16:04:27

Sounds like a var, rather than the value it contains, is making its way into the checkpoint state.

michaeldrogalis16:04:02

Yeah. I’m just surprised Nippy dealt with it to begin with.

michaeldrogalis16:04:02

Oh - your message says Unfreezable, not unthawable. Okay this makes more sense.

michaeldrogalis16:04:14

(def x 1) (nippy/freeze #'x) => Unfreezable type: class clojure.lang.Var

kjothen17:04:13

@michaeldrogalis @lucasbradstreet I would agree, but it is odd that it only happens when I have a punctuation trigger: a segment trigger works fine with the checkpointing of state, and both triggers use the same emit function. Also, it seems to occur directly after the :job-competed event-type has passed through the pipeline. Anyhow, not much you can do without a reproducible use case, so I'll get on with that. Thanks!

michaeldrogalis17:04:56

@kjothen Okay, that’s good evidence for a bug around that trigger type then. Thanks, we’ll get it fixed after we get some more info!

michaeldrogalis18:04:17

@kjothen Great, thanks! Can you toss this into an issue on Github?

michaeldrogalis18:04:30

Would like to keep a history of it there instead of Slack.

kjothen18:04:40

@michaeldrogalis sure, just wasn't convinced it was an issue just yet, could entirely be my bad. But I'll post something just in case.

michaeldrogalis18:04:48

S’all good, at least we have a record of it now. 🙂

michaeldrogalis18:04:46

@kjothen Cool, reproduced. Taking a look now.

michaeldrogalis18:04:19

Yep, this is a bug on our side. I see what’s happening.

kjothen18:04:14

@michaeldrogalis that's super. So very keen to make use of the trigger emit functionality, but I do need to have processed all segments in a batch job into an aggregate for downstream, hence the punctuation trigger. Thanks for looking into it!

michaeldrogalis18:04:39

@kjothen Sure, no problem! I see the issue, deciding on what the most appropriate fix is..

michaeldrogalis19:04:09

It needs review & tests, but if you want to pull this down it should get you unstuck.

kjothen19:04:54

@michaeldrogalis magnificent, thanks again! I'll certainly be trying it out in anger tomorrow (100MM+ segments!)

kjothen19:04:48

Yep, that's just the way we write it in finance.

michaeldrogalis19:04:25

Ah, I’d seen that notation before but never knew what it meant. Cool. Well, should work like a charm. Let us know!