Fork me on GitHub

@lucasbradstreet a small question about plugin design rationale, i'm trying to figure out what's behind it. in kafka, you start the consumer inside the start plugin protocol function: in sqs, you start it outside the plugin protocol: what's the rationale behind it ? legacy artifact ? sqs being not restartable ?


i get the feeling that starting inside the start protocol function is the cleaner approach, and in sqs it only works because Amazon's async client is kind of like a big singleton that always persists. is this true ?


I find myself repeating a pattern when making stateful flows, where I want to only have the emitted state flow downstream, with the input segments to the stateful task not flowing to downstream tasks. Is there a setting or best practice to achieve this?


@dave.dixon :onyx/type :reduce will not emit transformed segments downstream


@dave.dixon with constraint that :trigger/emit must be set if it’s an intermediate node.


@lmergen to be honest it’s mostly a legacy artifact. I think it’d be better off in the start, but when I was converting over all the plugins I took some shortcuts since there was a lot of work to do.


@dave.dixon you can also use an :onyx/type :reduce on a terminal node and it won’t force you to use a plugin (though that’s not what you want here).


@dave.dixon We really need an onyx-example for :onyx/type :reduce. It was a recent addition and we have been busy with the business side.


@lucasbradstreet Many thanks. I must have read that section in the docs a thousand times and managed to miss that.


Do you think it needs fleshing out?


Probably. The explanation you gave above seems more clear.


There was a mistake in the doc anyway :onyx/task -> :onyx/type