Fork me on GitHub
#onyx
<
2016-04-11
>
zamaterian08:04:35

Hi, how do I configure the kafka writer to not propagate the Sentinel ?

lucasbradstreet08:04:58

What do you mean by propagate? I don't think the Kafka output plugin will ever write a sentinel to the output stream, though I guess I should check

zamaterian08:04:57

By propagate I mean that it writes the sentinel to the kafka/topic by the onyx.plugin.kafka/write-messages plugin. thx for the fast reply simple_smile

lucasbradstreet08:04:19

Hmm. Yeah, I don't think it should work like that. One second

lucasbradstreet08:04:00

Hmm indeed it does do that

lucasbradstreet08:04:33

I guess we depend on waiting for the done from our tests. Personally I'd like to remove writing the done from the output plugin completely. I'll have a chat to @michaeldrogalis about it later today. We'll either stop it from doing that or will add an option

jeroenvandijk09:04:18

@michael Was reading the checklist above. I’ve seen that Onyx doesn’t allow circular dependencies in the workflow. Is this related to not having the iterative option yet? (just wondering)

lucasbradstreet09:04:56

@jeroenvandijk: that's essentially the same issue, yes

jeroenvandijk09:04:17

@lucasbradstreet: Cool, good to know

otfrom13:04:44

michaeldrogalis: completely stealing ideas from onyx (and using ztellman's automat) to solve a similar, but slightly different problem. So, thanks for documenting things so well. Especially the Walkthrough.md

michaeldrogalis14:04:57

@otfrom: Glad to be of service. simple_smile

michaeldrogalis14:04:39

@zamaterian: Currently trying to get rid of the sentinel completely in the next release

michaeldrogalis14:04:39

@jeroenvandijk @lucasbradstreet I've been playing around with supporting iterative computation through flow conditions. Still need to come to a conclusion about well that expresses basic AI algorithms

michaeldrogalis15:04:55

The other option that I've been kicking around is supporting an :iterations key with each job that explicitly models looping-type behavior.

jeroenvandijk15:04:06

@michaeldrogalis: cool. I was thinking about it for the simulation use case where you use the output of an http request to create a new http request. That could be solved though by creating some extra (redudant) steps in a workflow (assuming the loop stops at some point)

michaeldrogalis15:04:43

@jeroenvandijk: That makes sense. For now I'd recommend that you use a side-channel input to emulate looping (e.g. an extra root in the graph), but long term we need to support it directly to make iterative algorithms performant.