@colinhicks I took a look through the code. Looks fantastic. Let me know when I should try to integrate it. 🙂

Sweet. Go for it – I just pushed my last tweak.

Excellent. 😄

@lucasbradstreet quick metrics question: the latency number for a task - from where to where does that measure?

also, is it possible for us to disambiguate onyx time from task time? that is, have a measure for just the segment transfer and a time for the task invocation?

batch latency measures the time for onyx/fn to be mapped over a batch

complete latency measures how long it takes for a segment to be read from an input source, until it has flowed through the whole job and been acked

so if we have variable batch sizes, the latency numbers can vary quite a bit

Yes, definitely. Batch latency is intended to give you an idea about how much latency for each individual segment, because of the whole batch. I think another metric that would be useful would be batch latency / batch size

btw, we added a discussion of all the metrics here https://github.com/onyx-platform/onyx-metrics#guide-to-types-of-metrics--diagnosing-issues

so does batch latency include any measure of inter-task segment transport, then?

@robert-stuttaford it’s kind of hard to measure individual segments, though the metric I described above might help

No, it’d be kinda hard to track, since we’d probably have to compare the clock time from one node with the clock time from another

complete latency is probably the best measure of the overall latency that you see for a single input segment

great, this helps a lot. will definitely give that doc a read

batch latency / batch size — how hard would this be to add? i think this would be tremendously useful to monitor

@greywolve is going to submit a PR 🙂

Fantastic 🙂

It won’t be hard, I just haven't gotten around to it.

thanks!

@lucasbradstreet bit of an interesting one:

clojure.lang.ExceptionInfo: Unfreezable type: class clojure.lang.Delay clojure.lang.Delay@5201a90
    as-str: "#object[clojure.lang.Delay 0x5201a90 {:status :pending, :val nil}]"
      type: clojure.lang.Delay
clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: Unfreezable type: class clojure.lang.Delay clojure.lang.Delay@5201a90
       as-str: "#object[clojure.lang.Delay 0x5201a90 {:status :pending, :val nil}]"
       job-id: #uuid "c332f7e4-40b8-4ba1-9bfe-d5e182745b1e"
     metadata: {:name "highstorm", :job-id #uuid "c332f7e4-40b8-4ba1-9bfe-d5e182745b1e", :job-hash "9a6a1b4369dc672dc9f41fb09fad4c6c51decfeac6fae387c03bf07dd35f"}
      peer-id: #uuid "4094fa21-4464-4def-baee-73b68c2a031f"
    task-name: :read-log
         type: clojure.lang.Delay

onyx and onyx-datomic “0.9.9"/“0.9.9.0"

@robert-stuttaford: no onyx/fn on the read-log task I assume? What version of the datomjc dependency are you using? I think you've seen this before...

even if we’re doing something silly, i presume that it shouldn’t do this

{:onyx/name :read-log
 :onyx/plugin :onyx.plugin.datomic/read-log
 :onyx/type :input
 :onyx/medium :datomic
 :datomic/uri db-uri
 :onyx/batch-size 200
 :onyx/max-pending 2000
 :onyx/pending-timeout 60000
 :onyx/max-peers 1
 :checkpoint/key "highstorm-streaming"
 :checkpoint/force-reset? true}

Well, some values aren't serializable. If a task returns a value that isn't, we can either drop it or reboot and hope the problem is transient

Datomic "0.9.5344"

i remember we did have this before, a long time ago, and it was something you fixed in the read-log plugin

Ok, the datomic log API must be returning a delay from somewhere. Is it happening all of the time?

I don't think we ever tracked it down

seen it on 1/3 instances so far. checking the others

yeah it happened on all 3 instances

perhaps read-log can detect a Delay and return an alternate suitable value?

probably []

Possibly, yes, but I will want to know why the log API is returning it first. If you'd like, you can put an onyx fn on it now to filter it out

it’s this line, right?

https://github.com/onyx-platform/onyx-datomic/blob/0.9.x/src/onyx/plugin/datomic.clj#L309

i’ll ask in #datomic

Looks ok to me, but maybe the way it implements seq is leaking delays out

we’ll find out in ~6 hours 🙂

thanks Lucas!

No worries. If datomic was open source I'm sure we'd figure it out pretty quickly...

zing!

@lucasbradstreet just pasting this here for you because it might scroll past before you’re back at your desk

from @marshall in #datomic:

marshall [15:39]  
<@U0509NKGK> the use of seq would be my guess as well

marshall [15:56]  
<@U0509NKGK> looking again - there’s also a `take read-size` which could be doing it

Am I understanding this correctly - you've found a bug datomic?

that’s debatable 🙂

tx-range returns a sequence of transactions, and you're doing a take from them, and that could cause a delay to fall out...? Seems like a datomic bug to me. Or does it not return a collection like a vector or list, and the problem is a leaky conversion in the way you treat that log?

It does promise that each element is a map

my guess is it’s a leaky conversion

it may be due to the fact that Datomic is JVM tech, not just Clojure tech

It does technically say a "range" - I have no idea what a range would be in terms of "types" though.

Think about it, it seems like one of the elements that it returns must be a delay, which suggests to me that the sequencing isn't the problem, it's the fact that a delay is leaking out rather than the value. It probably doesn't have any elements and isn't getting derefed somewhere?

I'm pretty sure it's datomic's fault, whatever the case, because we don't use delay in onyx anywhere

yep. Marshall is elevating to the Datomic team

That's a first for us, heh.

more in #datomic 🙂

really just playing messanger now 🤓

probably simpler for Lucas to talk them directly?

@colinhicks I had a play around with onyx-gen-doc and tried it on onyx-kafka. Fantastic stuff. I think next steps will be to move the repo into onyx-platform, if you're cool with that, then get it on our automatic release process so we can put a jar out to Clojars.

Then one at a time, we can convert the plugins over.

Great. Just sent the transfer req to you

^^ @michaeldrogalis

Seems like the plugin doc conversion process is a great place for community contributions, too...

Great, I moved onyx-gen-doc into onyx-platform and invited you as a contributor with write access. 🙂