Fork me on GitHub
#onyx
<
2018-04-14
>
lmergen13:04:19

@lucasbradstreet a small question about plugin design rationale, i'm trying to figure out what's behind it. in kafka, you start the consumer inside the start plugin protocol function: https://github.com/onyx-platform/onyx-kafka/blob/0.12.x/src/onyx/plugin/kafka.clj#L134 in sqs, you start it outside the plugin protocol: https://github.com/onyx-platform/onyx-amazon-sqs/blob/0.12.x/src/onyx/plugin/sqs_input.clj#L94 what's the rationale behind it ? legacy artifact ? sqs being not restartable ?

lmergen13:04:53

i get the feeling that starting inside the start protocol function is the cleaner approach, and in sqs it only works because Amazon's async client is kind of like a big singleton that always persists. is this true ?

dave.dixon22:04:23

I find myself repeating a pattern when making stateful flows, where I want to only have the emitted state flow downstream, with the input segments to the stateful task not flowing to downstream tasks. Is there a setting or best practice to achieve this?

lucasbradstreet22:04:38

@dave.dixon :onyx/type :reduce will not emit transformed segments downstream

lucasbradstreet22:04:07

@dave.dixon with constraint that :trigger/emit must be set if it’s an intermediate node.

lucasbradstreet22:04:17

@lmergen to be honest it’s mostly a legacy artifact. I think it’d be better off in the start, but when I was converting over all the plugins I took some shortcuts since there was a lot of work to do.

lucasbradstreet22:04:54

@dave.dixon you can also use an :onyx/type :reduce on a terminal node and it won’t force you to use a plugin (though that’s not what you want here).

lucasbradstreet23:04:36

@dave.dixon We really need an onyx-example for :onyx/type :reduce. It was a recent addition and we have been busy with the business side.

dave.dixon23:04:54

@lucasbradstreet Many thanks. I must have read that section in the docs a thousand times and managed to miss that.

lucasbradstreet23:04:35

Do you think it needs fleshing out?

dave.dixon23:04:19

Probably. The explanation you gave above seems more clear.

lucasbradstreet23:04:39

There was a mistake in the doc anyway :onyx/task -> :onyx/type