This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-09-12
Channels
- # admin-announcements (3)
- # beginners (17)
- # boot (52)
- # braveandtrue (95)
- # cider (4)
- # cljs-dev (2)
- # clojars (118)
- # clojure (146)
- # clojure-art (4)
- # clojure-austin (1)
- # clojure-finland (20)
- # clojure-italy (33)
- # clojure-nl (1)
- # clojure-russia (49)
- # clojure-spec (136)
- # clojure-uk (28)
- # clojurescript (134)
- # clojutre (1)
- # conf-proposals (64)
- # cursive (3)
- # datomic (76)
- # hoplon (11)
- # ipfs (7)
- # jobs (1)
- # jobs-rus (1)
- # leiningen (4)
- # luminus (4)
- # mount (9)
- # om (34)
- # onyx (34)
- # proton (1)
- # re-frame (4)
- # reagent (35)
- # ring (2)
- # ring-swagger (6)
- # rum (15)
- # untangled (87)
@aengelberg So what's happening with metrics is this - ideally, we could like to know the latency for every single segment that crosses through an Onyx job. If that's 100,000,000 segments, in the best of worlds, we'd like to know the latency for every single one of them, because then we could do a full analysis with all the data. There's two problems though. 1 - monitoring systems generally aren't designed to handle huge amounts of load. And 2 - you'd be wasting a lot of time actually emitting the metrics instead of doing work in the Onyx job.
So to work around this, we locally collect latencies for a few seconds, then compute a couple of different metrics to squash them down into single values.
One metric that you found is the 99th percentile batch latency. This number represents the maximum latency that 99% of the segments were at least as fast as.
You can wiki quantile distributions for a full explanation.
If we were only reporting quantile latency, though, we'll be missing what's in that last 1%, which might be really really bad latencies, and we want to see if that's the case. So max batch latency is basically the highest latency number in the whole batch.
is there not a way to show the minimum latency?
You could emit that as a metric if you want, but I'd question it's utility.
@aengelberg What's your goal with obtaining a minimum?
Maybe to get a better understanding of the range of latencies, if the lowest percentile is 50%
Not hurting for that feature though.
Thanks for the explanation!
Okay - you're suspicious of bursty latencies, I see.
No problem, though. Gonna sleep, talk tomorrow.
Is there any way to configure how onyx prints exceptions? this isn't especially useful in aurora logs which don't seem to use ANSI coloring:
16-Sep-12 19:43:47 ip-192-168-64-135 WARN [onyx.metrics.riemann] - Lost riemann connection
[37mjava.lang.Thread.run[m [32m Thread.java: 745[m [37mjava.util.concurrent.ThreadPoolExecutor$Worker.run[m [32mThreadPoolExecutor.java: 617[m [37mjava.util.concurrent.ThreadPoolExecutor.runWorker[m [32mThreadPoolExecutor.java: 1142[m [37mjava.util.concurrent.FutureTask.run[m [32m FutureTask.java: 266[m [37m...[m [32m [m [33mclojure.core/binding-conveyor-fn/[1;33mfn[m [32m core.clj: 1938[m [33monyx.metrics.riemann/riemann-sender/[1;33mfn[m [32m riemann.clj: 53[m [33monyx.metrics.riemann/riemann-sender/fn/[1;33mfn[m [32m riemann.clj: 56[m [33mclojure.core/[1;33mderef[m [32m core.clj: 2232[m [37mcom.aphyr.riemann.client.MapPromise.deref[m [32m MapPromise.java: 62[m [37mcom.aphyr.riemann.client.MapPromise.unsafeDeref[m [32m MapPromise.java: 71[m [37mcom.aphyr.riemann.client.MapPromise.unsafeDeref[m [32m MapPromise.java: 77[m [37mcom.aphyr.riemann.client.MapPromise.mapCapturingExceptions[m [32m MapPromise.java: 26[m [37mcom.aphyr.riemann.client.MsgValidator.call[m [32m MsgValidator.java: 8[m [37mcom.aphyr.riemann.client.MsgValidator.call[m [32m MsgValidator.java: 11[m [1;31mcom.aphyr.riemann.client.ServerError[m: [3mConnection refused[m
Hi, I don't see README information on where to specify aws credentials for the onyx-amazon-s3 plugin
(https://github.com/onyx-platform/onyx-amazon-s3). What is the best practice to configure that?
@aaelony We like to use Aero https://github.com/juxt/aero
Thanks @gardnervickers, just what I was looking for 🙂
The aws-s3-sdk will automatically pull them though
does this code pick it up automatically in onyx? https://github.com/onyx-platform/onyx-amazon-s3/blob/0.9.x/src/onyx/plugin/s3_utils.clj#L17-L19
Constructs a new client to invoke service methods on Amazon S3. A credentials provider chain will be used that searches for credentials in this order:
Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
Java System Properties - aws.accessKeyId and aws.secretKey
Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
Instance Profile Credentials - delivered through the Amazon EC2 metadata service
I see the onyx.log say things like s3plugin: progress event. TRANSFER_FAILED_EVENT
and I think it might be due to the creds somehow.
@aaelony Haven't looked into doing that before. What's needed to use them?
I think it's just a "x-amz-server-side-encryption: AES256" addendum on the put command
@aaelony onyx-amazon-s3 will auto pick them up in all the usual ways that amazon allows. It autodetects it
so that includes env variables, ~/.aws/credentials, etc
not sure why you’re seeing those failures, and I can’t say much about the encryption stuff right now since I’m off to bed