This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-04-25
Channels
- # admin-announcements (5)
- # arachne (1)
- # beginners (29)
- # boot (36)
- # cider (110)
- # clara (1)
- # cljs-dev (3)
- # cljs-edn (14)
- # cljsrn (24)
- # clojure (63)
- # clojure-belgium (3)
- # clojure-dusseldorf (5)
- # clojure-greece (9)
- # clojure-russia (142)
- # clojure-sg (15)
- # clojure-uk (20)
- # clojurebridge (4)
- # clojurescript (58)
- # data-science (1)
- # datomic (37)
- # editors (2)
- # editors-rus (7)
- # emacs (1)
- # garden (31)
- # hoplon (3)
- # jobs-discuss (8)
- # keechma (86)
- # leiningen (1)
- # liberator (2)
- # mount (23)
- # off-topic (2)
- # om (18)
- # onyx (42)
- # planck (1)
- # quil (6)
- # re-frame (8)
- # reagent (3)
- # ring-swagger (1)
- # specter (4)
- # untangled (1)
Hey guys, my Onyx presentation video is up, check it out, https://www.youtube.com/watch?v=MoDai3IyvN0 thnx @michaeldrogalis for the drawings again! Boy I definitely needed a bluetooth remote
I watched that over the weekend, nice work. Can you remind me where the demo repo for the code is again please @bcambel
Great one!
Yes please. Looks like we're performing the checks in the wrong order
I'm noticing that before-batch
, after-read-batch
and after-batch
lifecycles are called, in this order, repeatedly, even when there are no segments anywhere near the workflow. Is this expected?
Yes, after the batch-timeout is hit, an empty batch will be processed. We will likely be changing the default behaviour here though. The current behaviour isn't the best approach performance wise as the pipeline could be processing an empty batch when actual data comes in.
Usually this isn't a concern since processing an empty batch is fast, but it would burn up more CPU as you decrease the batch timeout significantly
Well it does have some benefits e.g. metrics calls via life cycles can record when there are empty batches
The default batch timeout is 50ms
Not without making the batch timeout high. If you're doing something complex in the lifecycle could you just check whether the batch is empty? That's what we typically do
I’m starting and stopping jobs locally. The first start everything works, on the second start nothing seems to happen and I see in the Onyx log Not enough virtual peers have warmed up to start the task yet, backing off and trying again…
. Any idea what I could be doing wrong?
When you say stopping, do you mean killing?
Yeah i’m using (onyx.api/kill-job peer-config job-id)
Onyx dashboard shows a cross
Hmm. That is odd then.
Another weird thing i see is that (component/stop system)
doesn’t work anymore
hangs forever
On first glance the code looks OK
So you start up the environment, submit a job, kill the job, submit a new job?
yeah exactly
Are you testing this with the meetup job, or is it a new job that you’re writing?
a new job, i’ve sent you a link
@acron: There are a handful of advantages to having the task pipeline process empty batches in the same code path that it uses for batches with segments. If we completely ignored them, you'd have virtually no way of signaling to an outside process that your system is behaving healthy with respect to ingestion of messages, as @lucasbradstreet was saying. We'll probably have a switch in the next major release to toggle the behavior on/off.
@michaeldrogalis: :thumbsup:
I’m trying to generate a function in a lifecycle to use it in another intermediate function in my workflow. Should I do this via :onyx/params
?
I thought i’ve seen an example, but I cannot find it now
If you’re generating a function, it’ll need to be injected into :onyx.core/params
via a lifecycle function.
This is because :onyx/params
in the task-map has to be serializeable
(defn inject-fn [{:keys [onyx.core/params onyx.core/task-map] :as event} lifecycle]
{:onyx.core/params (conj params (fn [x] myfn))})
ah cool
Something like that
Using {:lifecycle/before-task-start inject-fn} as a lifecycle-calls that you will refer to
Great, that works, thanks!