Fork me on GitHub
#onyx
<
2016-09-12
>
michaeldrogalis04:09:12

@aengelberg So what's happening with metrics is this - ideally, we could like to know the latency for every single segment that crosses through an Onyx job. If that's 100,000,000 segments, in the best of worlds, we'd like to know the latency for every single one of them, because then we could do a full analysis with all the data. There's two problems though. 1 - monitoring systems generally aren't designed to handle huge amounts of load. And 2 - you'd be wasting a lot of time actually emitting the metrics instead of doing work in the Onyx job.

michaeldrogalis04:09:41

So to work around this, we locally collect latencies for a few seconds, then compute a couple of different metrics to squash them down into single values.

michaeldrogalis04:09:18

One metric that you found is the 99th percentile batch latency. This number represents the maximum latency that 99% of the segments were at least as fast as.

michaeldrogalis04:09:30

You can wiki quantile distributions for a full explanation.

michaeldrogalis04:09:09

If we were only reporting quantile latency, though, we'll be missing what's in that last 1%, which might be really really bad latencies, and we want to see if that's the case. So max batch latency is basically the highest latency number in the whole batch.

aengelberg04:09:12

is there not a way to show the minimum latency?

michaeldrogalis04:09:25

You could emit that as a metric if you want, but I'd question it's utility.

michaeldrogalis04:09:01

@aengelberg What's your goal with obtaining a minimum?

aengelberg04:09:08

Maybe to get a better understanding of the range of latencies, if the lowest percentile is 50%

aengelberg04:09:50

Not hurting for that feature though.

aengelberg04:09:57

Thanks for the explanation!

michaeldrogalis04:09:17

Okay - you're suspicious of bursty latencies, I see.

michaeldrogalis04:09:33

No problem, though. Gonna sleep, talk tomorrow.

aengelberg20:09:51

Is there any way to configure how onyx prints exceptions? this isn't especially useful in aurora logs which don't seem to use ANSI coloring:

16-Sep-12 19:43:47 ip-192-168-64-135 WARN [onyx.metrics.riemann] - Lost riemann connection
 java.lang.Thread.run  Thread.java: 745 java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 617 java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1142 java.util.concurrent.FutureTask.run  FutureTask.java: 266 ...   clojure.core/binding-conveyor-fn/fn  core.clj: 1938 onyx.metrics.riemann/riemann-sender/fn  riemann.clj: 53 onyx.metrics.riemann/riemann-sender/fn/fn  riemann.clj: 56 clojure.core/deref  core.clj: 2232 com.aphyr.riemann.client.MapPromise.deref  MapPromise.java: 62 com.aphyr.riemann.client.MapPromise.unsafeDeref  MapPromise.java: 71 com.aphyr.riemann.client.MapPromise.unsafeDeref  MapPromise.java: 77 com.aphyr.riemann.client.MapPromise.mapCapturingExceptions  MapPromise.java: 26 com.aphyr.riemann.client.MsgValidator.call  MsgValidator.java: 8 com.aphyr.riemann.client.MsgValidator.call  MsgValidator.java: 11 com.aphyr.riemann.client.ServerError: Connection refused

aaelony21:09:57

Hi, I don't see README information on where to specify aws credentials for the onyx-amazon-s3 plugin (https://github.com/onyx-platform/onyx-amazon-s3). What is the best practice to configure that?

aaelony21:09:14

Thanks @gardnervickers, just what I was looking for 🙂

gardnervickers21:09:23

The aws-s3-sdk will automatically pull them though

gardnervickers21:09:30

Constructs a new client to invoke service methods on Amazon S3. A credentials provider chain will be used that searches for credentials in this order:
Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
Java System Properties - aws.accessKeyId and aws.secretKey
Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
Instance Profile Credentials - delivered through the Amazon EC2 metadata service

aaelony21:09:16

I thought it would, but I am looking into why it doesn't just yet

aaelony21:09:18

I see the onyx.log say things like s3plugin: progress event. TRANSFER_FAILED_EVENT and I think it might be due to the creds somehow.

aaelony21:09:50

is there a way to support AES256 encrypted buckets?

michaeldrogalis21:09:27

@aaelony Haven't looked into doing that before. What's needed to use them?

aaelony21:09:12

I think it's just a "x-amz-server-side-encryption: AES256" addendum on the put command

aaelony21:09:51

"s3:x-amz-server-side-encryption": "AES256"

lucasbradstreet21:09:11

@aaelony onyx-amazon-s3 will auto pick them up in all the usual ways that amazon allows. It autodetects it

lucasbradstreet21:09:30

so that includes env variables, ~/.aws/credentials, etc

lucasbradstreet21:09:11

not sure why you’re seeing those failures, and I can’t say much about the encryption stuff right now since I’m off to bed

aaelony21:09:17

ok, will try to get a more detailed error

aaelony21:09:09

thanks 🙂