Fork me on GitHub
#onyx
<
2019-03-05
>
ho0man08:03:20

Hi @lmergen, and thanks for your reply No not deduplication, actually each stream contains different info but that info may be about the same thing I am getting stock market info from different sources (scrapping some, receiving others from rabbitmq, others comming from different source through tcp socket, etc) And the system is supposed to compute some financial indicators (that may need data from these different sources)

ho0man08:03:19

I wanted to merge related segments from different sources and then send them down stream

lmergen08:03:46

that's a lot easier 🙂

ho0man08:03:04

:)) Yeah What I intend to do is use state management and triggers

ho0man08:03:26

Dunno if that’s the right way, though

lmergen08:03:04

so just for my understanding, say you get event X from stream A, you want to find related event X' from stream B and send those downstream ?

lmergen08:03:55

and if no related events are found, none are send downstream ?

lmergen08:03:55

sounds almost like a kind of windowing

lmergen08:03:05

where you cache all events in the window

ho0man08:03:50

Yeah, so that is the appropriate way, then ?

lmergen08:03:51

well maybe you can model it as just an aggregate as well

lmergen08:03:03

depends on your time window an exact requirements

lmergen08:03:27

but what i would do is use onyx' :aggregation keys

ho0man08:03:37

Hmm ... what did you mean by aggregarate (am I missin something) ?

ho0man08:03:53

Hmmm thanks alot @lmergen

lmergen08:03:04

well it's actually more like a reducer, that emits events only when it was found before

👍 1
ho0man08:03:54

About the trigger + window solution I have another problem : But trigger/pred doesn’t appear to be getting the segment How am I supposed to decide when to send the event down stream ?

lmergen08:03:06

yeah i think a window might not be the best abstraction here, because it only allows you to flush down the entire window state (i think, i do not have extensive experience with onyx + windows)

lmergen08:03:48

the reduction/aggregation as seen in the code sample above is better, because it allows you to separate the state (seen events) from what you want to emit (similar events that have been seen before)

ho0man08:03:08

Thanks a lot Will check it right away

roland18:03:01

I'd use a session window, triggering on segment count = 2

roland18:03:53

We had a similar problem but we needed to send downstream even if some sources were missing and we used session window + time gap trigger