Is it possible to set a window on an output task?

My goal is to keep track of how many segments have fully propagated from the input to the output, then fire a trigger if that reaches a certain number of segments.

Yes, that is possible

You can also set onyx/fns on input and outputs

This might be a dumb question or hard to answer; We set up metrics/monitoring with the grafana/riemann/influx stack and setup queries pretty much the same as in the onyx-benchmark project. The question is I have no idea what we are really looking at like what is good/bad?

The two I monitor most are complete latency (the time it takes for a segments to travel through the entire DAG and have all its child segments acked), and retry count

If the complete latency is about what I expect, and the retry count is zero, then things look good

If not, then you need to start drilling in to things

thanks, will give that a shot. Hard part is, I don’t really know what to expect, lol

Oh, the other thing to look out for is the pending count.

yeah, what exactly does that mean?

This will only be measured on your input tasks. Basically it’ll give you an idea of how many segments from the input task are in flight at any time

in my case i am reading from kafka

Right, in the case of kafka, it’ll probably get refilled pretty quickly anyway, so it may just sit around 10000 (default pending-size), and that is OK

ah ok, i was thinking it might mean what is left to be pulled from kafka

10K is the number i am seeing

It’s hard to advise what numbers to look out for, because many of them will depend a lot on your problem

The only unequivocally bad one is message retries

yeah, i totally get that

thats why i wasn’t sure how much help i could get on this, lol

If your retry count is 0 then it becomes more of a question of whether the complete latency / throughput is good enough for your needs. If it isn’t then you should start drilling down into what you can optimise

ok, that definitely helps out

I really hope to be able to bring you guys on at some point for some kind of consultation

is it MAX_complete_latency ? or one of the precentiles thats better to look at?

I usually just look at max (basically 100th percentile), because it’s calculated over X time period (10s I think?). It depends on what kind of numbers you’re trying to hit though.

ok cool

RE: consultation, I think performance tuning / going to production is a good time to get us to help a bit. Let us know if you’re ever close to pulling the trigger 🙂

I will, believe me!

thanks for everything you guys do as it is

You’re welcome 🙂

@lucasbradstreet: If I set a window on an output task and hook a trigger up to it (say, with a :segment trigger type), is that trigger guaranteed to sync after the segment has completed its operation in the output task?

A different question: Do I not need to worry about using :onyx/uniqueness-key if I know that I'll have unique records coming in through the input? I want to protect against the window counting a record twice that was inputted once (due to network failure), but I'm not sure if I need the uniqueness key to guarantee that.