This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-08-17
Channels
- # arachne (1)
- # beginners (42)
- # boot (4)
- # cider (28)
- # clara (9)
- # cljs-dev (149)
- # cljsrn (5)
- # clojure (185)
- # clojure-austin (2)
- # clojure-dusseldorf (4)
- # clojure-italy (14)
- # clojure-norway (1)
- # clojure-russia (18)
- # clojure-spec (35)
- # clojure-uk (36)
- # clojurescript (78)
- # core-async (6)
- # data-science (20)
- # datomic (48)
- # emacs (1)
- # fulcro (2)
- # garden (4)
- # hoplon (47)
- # jobs (5)
- # jobs-rus (1)
- # leiningen (2)
- # lumo (12)
- # off-topic (8)
- # om (8)
- # onyx (39)
- # parinfer (19)
- # re-frame (100)
- # reagent (15)
- # ring-swagger (1)
- # sql (8)
- # vim (1)
- # yada (20)
How would collecting metrics (say for prometheus) work in onyx? Would you simply inject the registry with a lifecycle? Would that registry be central across peers or would each peer report their own metrics?
note, this could include timing metrics, where you'd like to potentially see the input-through-to-output duration
Works with Prometheus out of the box. 🙂
I’d like a task to only send segments downstream that haven’t been sent before. Since this involves state and keeping track of what has been sent (by hash), my initial thought was to use a global window with a trigger that emits. Is this a good way to go about it, or perhaps there is a more simple solution? I’m using an in-memory non-distributed env for Onyx, but I do want to future proof it for moving to a distributed env.
@wildermuthn That’s sensible, yes. Global window + :trigger/emit
.
Ok cool.
@michaeldrogalis are there any good grafana dashboards floating around ( Prometheus ds )
@camechis With respect to Onyx core?
I don’t have a recent one on hand.
it looks like if there's an exception in a job, onyx automatically logs it as WARN. is there a way to configure it?
asking because I need it to be logged as ERROR in order for it to show up in Google's Stackdriver Error Reporting. i can override in a custom timbre output-fn but i'm wondering if i should.
It’s warn level in 0.10.0 too https://github.com/onyx-platform/onyx/blob/0.10.x/src/onyx/peer/task_lifecycle.clj#L131
Hmm. We felt like it should be a warning because it can be normal for it to reboot the peer, and come back up.
That’s an interesting idea.
lifecycle/handle-exception
You can set a lifecycle for task :all
so that you don’t have to set one for each task.
See the cheat sheet.
hm, wonder if any exception should log as error
?
- if there's an exception, i want an alert
- if an exception is benign, i should catch/handle it
alternatively: let the user configure it
“if there’s an exception, i want an alert” This seems very specific to your situation with only being able to forward that level to Google Error Reporting.
i.e. if there's an exception in a task, i want to know about it, so log it at the error level
@lucasbradstreet Right now, I'm using this to control error / exception processing for all of the tasks in my job. How would I convert this to use :all
instead?:
(defn handle-error? [event old-segment ex-obj all-new]
(instance? java.lang.Exception ex-obj))
(def no-error? (complement handle-error?))
(defn transform-error [event segment exception-obj]
(let [error-segment {:body
{:segment segment
:ex {:class (.getName (.getClass exception-obj))
:message (.getMessage exception-obj)
:stack-trace (with-out-str (print-cause-trace exception-obj))}}}]
(log/error error-segment)
error-segment))
(defn build-error-flow-cond [error-task source-task]
{:flow/from source-task
:flow/to [error-task]
:flow/predicate ::handle-error?
:flow/short-circuit? true
:flow/post-transform ::transform-error
:flow/thrown-exception? true})
(defn build-standard-error-flow-cond [source-task dest-task]
{:flow/from source-task
:flow/to [dest-task]
:flow/predicate ::no-error?
:flow/short-circuit? true
:flow/thrown-exception? false})
It looks like the :all
keyword is only available on :flow/to
and :flow/predicate-errors-to
. Why isn't is on the :flow/from
?
@stephenmhopper running out for a bit. Can help you when I get back.
yeah, no rush at all
The above code works great, but if there's a simpler way (and it sounds like there might be based on the discussion above), I'd love to know it