onyx 2018-06-26

Can i kill submit job pod in kubernetes after submitting the job?

I'm currently playing around with Onyx in the context of a small hobby project and I ran into an issue the other day, which I posted to SO: https://stackoverflow.com/questions/51038280/onyx-cant-pick-up-trigger-emit-results-in-the-next-task

Basically, I wanted to understand aggregate some data and hand it to the next task and thought :trigger/emit would be the way to go and then the next task never saw the emitted segments.

Now, I finally saw that there was a warning / error in the log file that pointed to a missing peer configuration of :onyx.peer/storage.zk.insanely-allow-windowing?

While this now makes my toy example working, this option (and the corresponding commit in Onyx) doesn't look like this is a good idea.

But what is it, then?

If the only reason it didn’t work was :onyx.peer/storage.zk.insanely-allow-windowing?, then that means everything else around :trigger/emit was ok.

I would just recommend switching to S3 checkpointing, as you will have trouble with ZK based window checkpoints.

👍 1

@sreekanth if it’s just a pod that calls submit-job, then yes

👍 1