Fork me on GitHub

@michaeldrogalis Hey! I have created a repo with minimal reproduction of the problem with lazy seqs we discussed yesterday. Here’s the issue


This is probably a simple question: I’m hitting a ‘unable to resolve symbol’ when trying to run a workflow in test. Right now the job is spun up in test. It contains a catalog entry that refer’s to a function from a namespace in the main codebase. Since that function is only represented as a catalog entry I don’t refer/import it anywhere directly in the test namespace. When the job runs that fn can’t be found. How do I ensure that it and others like it can be resolved?


So, it resolves it if I explicitly require the function in the enclosing namespace that runs the job. Is there any other way to ensure it can be found?


@jetmind Thanks! I’ll get a look at it tonight.


@georgek That’s about it. It’s the same way as in Clojure. To ensure something is resolvable on the classpath, you need to require it.


Kewl, that’s what I suspected. 🙂


I have hit a problem with punctuation triggers working on 0.10.beta9. From what I can tell, onyx.peer.task-lifecycle/checkpoint-state throws an "Unfreezable type: class clojure.lang.Var" exception, possibly because it's trying to serialize the trigger's :pred-fn? Exception is thrown with any predicate function I use, even (constantly false). Not entirely sure what to do next, other than remove the trigger entirely and workaround it using another trigger strategy.


@kjothen Can you upgrade to beta12 and try to reproduce it?


@kjothen I’ll take a look, that’s a perplexing one. We don’t serialize the pred-fn in the checkpoints


If you can Gist us the full stack trace, that’d be appreciated. Also your window & trigger definition.


@michaeldrogalis Same behaviour under beta12 too. The :pred-fn was just a (bad) guess, I'll prepare the gist (which is tricky because my work PC cannot access slack or Gist!) If it helps, I have a global window using the built in conj aggregator and the (new!) emit trigger work. I'm only using the punctuation trigger because I need to ensure the window state is emitted downstream before the job competes.


@kjothen If the stack trace blew up at onyx.peer.task-lifecycle/checkpoint-state, you probably have something in your Conj aggregate that can’t be deserialized.


I’m not sure how Nippy was able to freeze something it couldn’t thaw, though.


What types of values are you putting into the window?


Sounds like a var, rather than the value it contains, is making its way into the checkpoint state.


Yeah. I’m just surprised Nippy dealt with it to begin with.


Oh - your message says Unfreezable, not unthawable. Okay this makes more sense.


(def x 1) (nippy/freeze #'x) => Unfreezable type: class clojure.lang.Var


@michaeldrogalis @lucasbradstreet I would agree, but it is odd that it only happens when I have a punctuation trigger: a segment trigger works fine with the checkpointing of state, and both triggers use the same emit function. Also, it seems to occur directly after the :job-competed event-type has passed through the pipeline. Anyhow, not much you can do without a reproducible use case, so I'll get on with that. Thanks!


@kjothen Okay, that’s good evidence for a bug around that trigger type then. Thanks, we’ll get it fixed after we get some more info!


@kjothen Great, thanks! Can you toss this into an issue on Github?


Would like to keep a history of it there instead of Slack.


@michaeldrogalis sure, just wasn't convinced it was an issue just yet, could entirely be my bad. But I'll post something just in case.


S’all good, at least we have a record of it now. 🙂


@kjothen Cool, reproduced. Taking a look now.


Yep, this is a bug on our side. I see what’s happening.


@michaeldrogalis that's super. So very keen to make use of the trigger emit functionality, but I do need to have processed all segments in a batch job into an aggregate for downstream, hence the punctuation trigger. Thanks for looking into it!


@kjothen Sure, no problem! I see the issue, deciding on what the most appropriate fix is..


It needs review & tests, but if you want to pull this down it should get you unstuck.


@michaeldrogalis magnificent, thanks again! I'll certainly be trying it out in anger tomorrow (100MM+ segments!)


Yep, that's just the way we write it in finance.


Ah, I’d seen that notation before but never knew what it meant. Cool. Well, should work like a charm. Let us know!