@michaeldrogalis: nice - is that focusing more on the EL or is there going to be a something for the T bit too ?

@mccraigmccraig: Will add more T as needed. Need to be careful not to try and rebuild Onyx into a tool, so err'ing on the side of less transformational power.

nice @michaeldrogalis !

see questions about this sql->datomic all the time

@lucasbradstreet, sorry to bother, but i'm trying to debug some metrics issues we're having. the

:complete-latency

Are you using the latest version?

Actually I lie

It was throughput

Are the metrics on an input task?

yup we are

yeah i have it set to just the one input task now

And only that metric?

Do you get retries?

yup, batch, retries, and throughput seem fine

it's like its not being put on the channel

That is super weird

haha i know, i've been at this all day

I assume your test is long enough to run for 10 seconds?

finally decided to ask for help

😛

yeah, i ran it for a minute at a time

I'm out grabbing some food but can help when I get back

It seems to pass the event

Err test

https://github.com/onyx-platform/onyx-metrics/blob/0.8.x/test/onyx/metrics/send_test.clj

That checks whether events with the complete latency tags were sent

thanks! hmm interesting

Hmm. Are you getting non zero retries?

Maybe it's not completing any message so it's always nil and never puts the event

Ah yes, the value in the log entries you posted have :value nil

Hmm. Although we do still seem to put them on the channel

oh yeah, but there are entries that aren't nil, and they still don't get sent, also there doesn't appear to be anything stopping it from putting if the value is nil? actually the read-log (an input task) doesn't seem to be working at all. it outputs throughput, and batch once and then after that it doesn't do anything

Yeah, was a false track

Try running the send test with your logging still in there

kk

going to try the send-test in a sec

(the above values are being logged from the send-fn thread, but notice there's still no complete latency)

(that's for our input task)

Yeah it should be with the retry rate

Actually retry rate should be within the when that checks if it's an input task

Though it should still log every second

doesn't seem to print anything when i run the test

but it passes

Checking the logs with grep, not eyeballing right?

It's a real puzzler

Nothing in onyx.log?

oh right, silly me, lol, let me try that again

but i guess that's enough for the test

That's ok because the test only lasts around 10a

10s

Retry is sampled every second. Complete latency every 10s

My current suspicion is that it's outputting it in your main code, but only 1/10th of the time you see it with a retry so you may be missing it

you're right the problem seems to be in my code, it looks like its a silent assertion error, which doesn't pop up anywhere, causing it, on one of my fns which transforms the metric to a statsd compatible one

a super simple sender fn loop works fine

pretty annoying that the assertions are just ignored

that's the whole point of having them there

haha

Ok, I was going to say you may want to be careful in your future thread to catch exceptions

Ok, I was going to say you may want to be careful in your future thread to catch exceptions

But you said it didn't happen in timbre which confused matters :)

yeah i missed it because it only happens 1/10th of the time like you said, lesson learned, always grep 😛 or log so only that can appear

i guess i need to learn more about using futures 😛

ahh, the docs

haha

oh well, at least i know my narrowing things ability is decent, haha

sorry for wasting your time lucas

Glad it’s working, and looking forward to the PR :d

😄

happy to PR if you want to add in built in metrics for datadog flavoured statsd? not sure if you guys want that 😛 if you don't we'd probably release it as an onyx metrics plugin on its own

what should one send when the value of the metric is nil?

@greywolve: I think the thing to do is omit sending the metric. Most of these dashboards have configuration that lets you pick what to display when a metric is missing for a time interval. Correct me if I'm wrong though.

ta, i'll do that then

I would say “no value” for completion latency, not 0

it really depends on the metrics you’re calculating