Fork me on GitHub
#onyx
<
2017-08-17
>
dadair14:08:37

How would collecting metrics (say for prometheus) work in onyx? Would you simply inject the registry with a lifecycle? Would that registry be central across peers or would each peer report their own metrics?

dadair14:08:07

note, this could include timing metrics, where you'd like to potentially see the input-through-to-output duration

michaeldrogalis15:08:34

Works with Prometheus out of the box. 🙂

ghadi17:08:20

I ❤️ prometheus

wildermuthn16:08:00

I’d like a task to only send segments downstream that haven’t been sent before. Since this involves state and keeping track of what has been sent (by hash), my initial thought was to use a global window with a trigger that emits. Is this a good way to go about it, or perhaps there is a more simple solution? I’m using an in-memory non-distributed env for Onyx, but I do want to future proof it for moving to a distributed env.

michaeldrogalis16:08:41

@wildermuthn That’s sensible, yes. Global window + :trigger/emit.

Travis16:08:07

@michaeldrogalis are there any good grafana dashboards floating around ( Prometheus ds )

michaeldrogalis16:08:23

@camechis With respect to Onyx core?

Travis16:08:19

Just some dashboards to monitor metrics from tasks, just curious

michaeldrogalis16:08:53

I don’t have a recent one on hand.

Travis16:08:01

No worries

devth16:08:46

it looks like if there's an exception in a job, onyx automatically logs it as WARN. is there a way to configure it?

devth16:08:56

trying to hunt down the source where it'd be logged

devth16:08:37

asking because I need it to be logged as ERROR in order for it to show up in Google's Stackdriver Error Reporting. i can override in a custom timbre output-fn but i'm wondering if i should.

lucasbradstreet17:08:59

Hmm. We felt like it should be a warning because it can be normal for it to reboot the peer, and come back up.

devth17:08:11

ah. i think i will wrap and throw inside my tasks at the error level for now.

devth17:08:46

maybe warn when (= exception-action :restart) but error otherwise?

lucasbradstreet18:08:27

That’s an interesting idea.

devth19:08:50

btw what decides whether a task is restarted in 0.9.x when it throws?

lucasbradstreet19:08:28

lifecycle/handle-exception

devth19:08:49

thanks. does that apply to 0.10.x too?

devth19:08:15

cool. not familiar - i'll have to read up

lucasbradstreet19:08:18

You can set a lifecycle for task :all so that you don’t have to set one for each task.

lucasbradstreet19:08:23

See the cheat sheet.

devth19:08:28

great, thanks

devth19:08:38

hm, wonder if any exception should log as error? - if there's an exception, i want an alert - if an exception is benign, i should catch/handle it alternatively: let the user configure it

michaeldrogalis20:08:43

“if there’s an exception, i want an alert” This seems very specific to your situation with only being able to forward that level to Google Error Reporting.

devth20:08:11

i think it's pretty standard for error level logging to trigger alerts

devth20:08:31

i.e. if there's an exception in a task, i want to know about it, so log it at the error level

stephenmhopper20:08:00

@lucasbradstreet Right now, I'm using this to control error / exception processing for all of the tasks in my job. How would I convert this to use :all instead?:

(defn handle-error? [event old-segment ex-obj all-new]
  (instance? java.lang.Exception ex-obj))

(def no-error? (complement handle-error?))

(defn transform-error [event segment exception-obj]
  (let [error-segment {:body
                       {:segment segment
                        :ex {:class (.getName (.getClass exception-obj))
                             :message (.getMessage exception-obj)
                             :stack-trace (with-out-str (print-cause-trace exception-obj))}}}]
    (log/error error-segment)
    error-segment))

(defn build-error-flow-cond [error-task source-task]
  {:flow/from source-task
   :flow/to [error-task]
   :flow/predicate ::handle-error?
   :flow/short-circuit? true
   :flow/post-transform ::transform-error
   :flow/thrown-exception? true})

(defn build-standard-error-flow-cond [source-task dest-task]
  {:flow/from source-task
   :flow/to [dest-task]
   :flow/predicate ::no-error?
   :flow/short-circuit? true
   :flow/thrown-exception? false})

stephenmhopper20:08:39

It looks like the :all keyword is only available on :flow/to and :flow/predicate-errors-to. Why isn't is on the :flow/from?

lucasbradstreet20:08:40

@stephenmhopper running out for a bit. Can help you when I get back.

stephenmhopper20:08:46

yeah, no rush at all

stephenmhopper20:08:25

The above code works great, but if there's a simpler way (and it sounds like there might be based on the discussion above), I'd love to know it