@michaeldrogalis it's a single task that accepts a list of segments (:onyx/batch-fn? true). my plan was to have flow conditions from the task in question, one routing segments that were successul, another routing segments to an error task (leaf task that just logs the error). i couldn't work out what to do with the messages i wanted to retry (since the workflow graph has to be acyclic - is there a way to tell a segment to go back to the beginning?).

dear all, I was wondering whether someone knows if the onyx-kafka plugin version 0.11.x works or works not with kafka version 0.9? i have tested on a kafka cluster with both version 0.9 and 0.11. On the latter it seems to work and the first there is a timeout on fetching metadata

Ah I think I can answer my own question from: https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

@ben.mumford There used to be in versions of Onyx prior to 0.10 - but we removed that because of the way the streaming engine works. Routing back to the beginning would cause segments after it to replay again as well.

IMO the best option in that case would be putting those segments back into the input source.

Partial retries are admittedly a thorny issue. There’s a lot of pitfalls in every layer.

It’s not so much of an Onyx-specific issue as it is controlling how fault recovery works in that case. The options aren’t good across the board.

@eelke Likely only 0.10 and beyond - 0.9 and 0.10 are close in terms of API compatibility but not a perfect match.