This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-01-20
Channels
- # arachne (11)
- # aws (2)
- # beginners (33)
- # boot (167)
- # cider (71)
- # clara (2)
- # cljs-dev (28)
- # cljsrn (3)
- # clojars (1)
- # clojure (83)
- # clojure-austin (21)
- # clojure-dev (24)
- # clojure-russia (19)
- # clojure-spec (33)
- # clojure-uk (108)
- # clojurescript (114)
- # component (1)
- # core-async (1)
- # cursive (7)
- # datomic (13)
- # editors (1)
- # emacs (15)
- # hoplon (10)
- # lein-figwheel (4)
- # leiningen (3)
- # mount (2)
- # om (134)
- # om-next (4)
- # onyx (42)
- # pedestal (41)
- # quil (2)
- # re-frame (29)
- # reagent (4)
- # remote-jobs (6)
- # ring-swagger (5)
- # untangled (9)
I think onyx-dashboard needs version indication somewhere, given that upgrading onyx requires upgrading onyx-dashboard 🙂
@asolovyov yeah, it definitely does need that
this problem is really weird:
:read-mk-campaign logging segment: {:offset 3030
:read-mk-campaign logging segment: {:offset 3031
:read-mk-campaign logging segment: {:offset 3032
:read-mk-campaign logging segment: {:offset 3033
:read-mk-campaign logging segment: {:offset 3034
:read-mk-campaign logging segment: {:offset 3035
:read-mk-campaign logging segment: {:offset 3028
:read-mk-campaign logging segment: {:offset 3037
:read-mk-campaign logging segment: {:offset 3033
:read-mk-campaign logging segment: {:offset 3032
:read-mk-campaign logging segment: {:offset 3031
:read-mk-campaign logging segment: {:offset 3030
:read-mk-campaign logging segment: {:offset 3029
:read-mk-campaign logging segment: {:offset 3036
:read-mk-campaign logging segment: {:offset 3035
:read-mk-campaign logging segment: {:offset 3034
and it seems it goes through whole task, actually writes feedback to kafka, and then starts again
@lucasbradstreet should it write something in logs on retry? because it seems there are some retries, it's just a bit lower than other charts and harder to notice 🙂
Ah, no, it will not write anything to the timbre logs
It’s usually one of two things. Either the segment is causing a task to crash (and thus reboot), or those segments are taking longer than onyx/pending-timeout to process
I think pending-timeout is 180 seconds by default, which is pretty long for most use cases but maybe this is a longer one
reducing onyx/max-pending can help a lot too
try reducing it to something really low, and see if things process successfully (possibly slowly) and then start to increase it again
when you have a big max-pending, things will queue up so you can end up with some segments taking longer than pending-timeout even though they shouldn’t take all that long
Sure, try raising pending-timeout. Also check your logs to see if you’re seeing any exceptions
java.nio.file.NoSuchFileException: /dev/shm/aeron-rc11400 file: "/dev/shm/aeron-rc11400" clojure.lang.ExceptionInfo: Error in component :messaging-group in system onyx.system.OnyxPeerGroup calling #'com.stuartsierra.component/stop component: #<Aeron Peer Group> function: #'com.stuartsierra.component/stop reason: :com.stuartsierra.component/component-function-threw-exception system: #<Onyx Peer Group> system-key: :messaging-group
the shutdown-peer-group is called when mount is stopping the application when its being closed
any idea what that means (i get that no file exists) but why does it only do happen some of the times (when not stopping from a repl)
@lucasbradstreet is there a way to determine why it's retrying? 🙂 because when I run it locally, I see the time difference between reads from kafka is a minute
like that:
17-01-20 10:00:49 alcor.mk.corp DEBUG [raker.common.logging:16] - :read-mk-campaign logging segment: {:offset 624,
17-01-20 10:00:49 alcor.mk.corp DEBUG [raker.common.logging:16] - :read-mk-campaign logging segment: {:offset 625,
17-01-20 10:01:49 alcor.mk.corp DEBUG [raker.common.logging:16] - :read-mk-campaign logging segment: {:offset 625,
17-01-20 10:01:49 alcor.mk.corp DEBUG [raker.common.logging:16] - :read-mk-campaign logging segment: {:offset 624,
@lucasbradstreet thanks a lot, in the end it was pending timeout - it's 60 seconds by default and turns out my task takes from 3 to 4 minutes :))
@asolovyov: great. Glad to hear it's something standard. You might find complete latency a useful metric to track in addition to retries. Complete latency will tell you how long a message took to ack end to end.
I'm tracking it, it's just when retry happens, complete latency is not sent to riemann
Please review and merge new dashboard UI PR- https://github.com/onyx-platform/onyx-dashboard/pull/79
Thanks @mariusz_jachimowicz. I’ll try to get a look at it this weekend. Much appreciated.