This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-04-06
Channels
- # admin-announcements (17)
- # beginners (78)
- # boot (162)
- # braid-chat (2)
- # cider (20)
- # cljs-dev (9)
- # cljsjs (41)
- # cljsrn (17)
- # clojure (98)
- # clojure-austin (5)
- # clojure-brasil (1)
- # clojure-dusseldorf (1)
- # clojure-greece (1)
- # clojure-ireland (2)
- # clojure-italy (1)
- # clojure-japan (5)
- # clojure-russia (128)
- # clojure-uk (2)
- # clojurescript (29)
- # core-async (1)
- # core-logic (7)
- # css (1)
- # cursive (12)
- # datomic (18)
- # devcards (1)
- # dirac (6)
- # emacs (31)
- # funcool (28)
- # hoplon (208)
- # jaunt (66)
- # jobs (1)
- # juxt (6)
- # lein-figwheel (14)
- # off-topic (9)
- # om (83)
- # om-next (6)
- # onyx (63)
- # overtone (1)
- # parinfer (2)
- # protorepl (23)
- # re-frame (27)
- # reagent (14)
- # ring-swagger (8)
- # slack-help (2)
- # spacemacs (1)
- # untangled (56)
;; Fire the trigger every 10 segments we see. This corresponds to the number ;; of items in our toy data set - which gives us the predictability of ;; knowing that it will fire 3 times. :trigger/threshold [10 :elements]
Hi leonfs. Go for it
Hmm yes, I can see how that is confusing. What’s probably happening is that it’s firing on task completion as well
the :onyx.triggers/segment trigger will also fire on job / task completion so that you can flush your windows even if you haven’t hit the threshold
That appears to be the case. We need to fix up the wording / case to be less confusing
(defn deliver-promise! [event window trigger {:keys [window-id lower-bound upper-bound event-type]} state]
(println "Event type was " event-type)
(let [lower (java.util.Date. lower-bound)
upper (java.util.Date. upper-bound)]
(swap! fired-window-state assoc [lower upper] (into #{} state))))
I pulled out the event type key and it seems to be triggered for "Event type was :job-completed"
Thanks. I created an issue https://github.com/onyx-platform/learn-onyx/issues/16
The :done being read on all input tasks, and the segments being acked through the whole workflow
i.e. all data has been processed
but in a production environment, let’s say input data coming from Kafka, then you can’t effectively know what “all” input is right?
Yes, that’s right. For those jobs there’d be no way to “complete” the job, only kill a long running job
Event type was :new-segment Event type was :new-segment Event type was :new-segment Event type was :job-completed Event type was :job-completed Event type was :job-completed
how can I see from which extent the delivery came from? I tried to use window-id but I got null
This documents the keys of the state-event http://www.onyxplatform.org/docs/cheat-sheet/latest/#/state-event
The extent maps directly to lower-bound and upper-bound
I think generally you only care about what the lower and upper bounds of the windows are, not their extents
cool.. then I would add that on github issue change the window-id key to window as window-id is not part of the state event
Please do - that’s a vestige of our old version. Now that we include the window you can just look it up there
You’re welcome
;; When set to true, if any "extent", an instance of a window, fires, all ;; extents will fire. This is desirable behavior for segment-based triggers, ;; so we turn it on. :trigger/fire-all-extents? true
Without it you will only see a window associated to your event fire e.g. last segment may end up in the 10:00-10:05 bucket, which would be the extent that fires when that trigger is hit.
Since we might want to flush all of our window buckets, when 10 segments come through, we need that option turned on, so that all of our buckets will have the trigger sync fn called on them
not just the one matching the last event
thanks Lucas.. I’ve been playing with the threshold and the input vector to test what you just mentioned
but for the given input, if the threshold is set 2 to elements, fire-all-extents to false.. shouldn’t (println "Event type was" event-type "on" (.format (java.text.SimpleDateFormat. "h") (java.util.Date. upper-bound))) be 4 instead of 3…
{:event-id 1 :event-time #inst "2015-11-20T03:15:00.000-00:00" :page-visited "/"} {:event-id 2 :event-time #inst "2015-11-20T04:45:00.000-00:00" :page-visited "/login"}
On first look, that does appear very odd.
I’m just about to have dinner, I’ll give it a better look when I’ve finished
Another question here guys.. Usually in the windowing examples the window points to a task that references an identity function. if for example the function would modify the segment would the window receive the recently transformed segment?
So windowing cannot emit window state downstream currently. So downstream would just recieve a segment transformed by your task function, in the case of the windowing examples, transformed by identity
The only way to get data out of your aggregations/windows is through the trigger :sync
@leonfs Changing this behavior to allow multiple serial aggregations is a focus right now
the only way to achieve NOW multiple serial aggregations would be to push the state of the window to another input source from :sync right?
something durable preferably
Some folks are using Amazon SQS for it right now and that’s working well, Kafka would be another good choice (and is similar to how Spark works)
That particular issue is being tracked here https://github.com/onyx-platform/onyx/issues/552 . Development is moving swiftly on it so I’m not sure that keeping the issue up to date has been a priority.
We do not currently have a roadmap but the need for one has been brought up.
cool.. thks gardner.. I will take a look to the issue.. it’s a fascinating field.. Hopefully once I’m up to date with the inner workings of Onyx I will be able to help/contribute to the project..
Awesome, seems like your moving quickly