Fork me on GitHub

Hi, how do I configure the kafka writer to not propagate the Sentinel ?


What do you mean by propagate? I don't think the Kafka output plugin will ever write a sentinel to the output stream, though I guess I should check


By propagate I mean that it writes the sentinel to the kafka/topic by the onyx.plugin.kafka/write-messages plugin. thx for the fast reply simple_smile


Hmm. Yeah, I don't think it should work like that. One second


Hmm indeed it does do that


I guess we depend on waiting for the done from our tests. Personally I'd like to remove writing the done from the output plugin completely. I'll have a chat to @michaeldrogalis about it later today. We'll either stop it from doing that or will add an option


@michael Was reading the checklist above. I’ve seen that Onyx doesn’t allow circular dependencies in the workflow. Is this related to not having the iterative option yet? (just wondering)


@jeroenvandijk: that's essentially the same issue, yes


michaeldrogalis: completely stealing ideas from onyx (and using ztellman's automat) to solve a similar, but slightly different problem. So, thanks for documenting things so well. Especially the


@otfrom: Glad to be of service. simple_smile


@zamaterian: Currently trying to get rid of the sentinel completely in the next release


@jeroenvandijk @lucasbradstreet I've been playing around with supporting iterative computation through flow conditions. Still need to come to a conclusion about well that expresses basic AI algorithms


The other option that I've been kicking around is supporting an :iterations key with each job that explicitly models looping-type behavior.


@michaeldrogalis: cool. I was thinking about it for the simulation use case where you use the output of an http request to create a new http request. That could be solved though by creating some extra (redudant) steps in a workflow (assuming the loop stops at some point)


@jeroenvandijk: That makes sense. For now I'd recommend that you use a side-channel input to emulate looping (e.g. an extra root in the graph), but long term we need to support it directly to make iterative algorithms performant.