Fork me on GitHub
#onyx
<
2018-09-20
>
Lutz07:09:03

Is there any documentation on how :reduce tasks work? Is there any difference between a terminal :reduce task and an terminal (i.e., :output) task with plugin :onyx.peer.function/function?

Lutz11:09:34

I have a workaround for above watermark triggering issue. I can explain it and maybe you can tell if that is a good idea or not. I know the incoming data are chronological. I want fixed windows of these data. (I also know that there are no empty windows). Watermark triggering does not work in my specific case, I know now. What I do to get the same behaviour, though, is the following: In an upstream task I emit an additional segment for each processed segment, where I include just a special keyword and a timestamp by <the same time of the original segment minus the window range> to make sure it will end up in the previous window. With a punctuation trigger I check if an incoming segment has the special keyword, and if yes, I trigger. Are there any ways in which this can break or is a bad idea otherwise? E.g., does this still work if you scale up the number of peers?

lucasbradstreet21:09:23

This approach will have problems if you scale up to more peers and don’t emit a segment that will reach each peer.

lucasbradstreet21:09:55

Which can be hard to do if you have a grouping-fn, and extremely difficult if not because the peer it’ll be sent to is randomised then.