Fork me on GitHub
#onyx
<
2016-04-25
>
bcambel08:04:58

Hey guys, my Onyx presentation video is up, check it out, https://www.youtube.com/watch?v=MoDai3IyvN0 thnx @michaeldrogalis for the drawings again! Boy I definitely needed a bluetooth remote

jasonbell10:04:55

I watched that over the weekend, nice work. Can you remind me where the demo repo for the code is again please @bcambel

lucasbradstreet10:04:47

Yes please. Looks like we're performing the checks in the wrong order

acron11:04:01

I'm noticing that before-batch, after-read-batch and after-batch lifecycles are called, in this order, repeatedly, even when there are no segments anywhere near the workflow. Is this expected?

lucasbradstreet11:04:42

Yes, after the batch-timeout is hit, an empty batch will be processed. We will likely be changing the default behaviour here though. The current behaviour isn't the best approach performance wise as the pipeline could be processing an empty batch when actual data comes in.

lucasbradstreet11:04:28

Usually this isn't a concern since processing an empty batch is fast, but it would burn up more CPU as you decrease the batch timeout significantly

acron11:04:25

Cool, thanks

acron11:04:44

If feels like it shouldn't be calling the lifecycle functions for empty batches, no?

acron11:04:28

Also, the rate at which we're seeing the calls is well in excess of the batch timeout

acron11:04:37

As in, multiple times a second

acron11:04:52

(With a batch timeout of 1000ms)

lucasbradstreet11:04:03

Well it does have some benefits e.g. metrics calls via life cycles can record when there are empty batches

lucasbradstreet11:04:12

The default batch timeout is 50ms

acron11:04:42

Aha, it is batch timeout, sorry you were correct.

acron11:04:36

Any way to disable batch timeouts? Bar a crazy high timeout?

lucasbradstreet11:04:15

Not without making the batch timeout high. If you're doing something complex in the lifecycle could you just check whether the batch is empty? That's what we typically do

jeroenvandijk12:04:33

I’m starting and stopping jobs locally. The first start everything works, on the second start nothing seems to happen and I see in the Onyx log Not enough virtual peers have warmed up to start the task yet, backing off and trying again…. Any idea what I could be doing wrong?

lucasbradstreet12:04:07

When you say stopping, do you mean killing?

jeroenvandijk12:04:30

Yeah i’m using (onyx.api/kill-job peer-config job-id)

jeroenvandijk12:04:55

Onyx dashboard shows a cross

lucasbradstreet12:04:00

Hmm. That is odd then.

jeroenvandijk12:04:10

Another weird thing i see is that (component/stop system) doesn’t work anymore

lucasbradstreet12:04:18

On first glance the code looks OK

lucasbradstreet12:04:01

So you start up the environment, submit a job, kill the job, submit a new job?

lucasbradstreet12:04:02

Are you testing this with the meetup job, or is it a new job that you’re writing?

jeroenvandijk12:04:48

a new job, i’ve sent you a link

michaeldrogalis14:04:52

@acron: There are a handful of advantages to having the task pipeline process empty batches in the same code path that it uses for batches with segments. If we completely ignored them, you'd have virtually no way of signaling to an outside process that your system is behaving healthy with respect to ingestion of messages, as @lucasbradstreet was saying. We'll probably have a switch in the next major release to toggle the behavior on/off.

jeroenvandijk17:04:15

I’m trying to generate a function in a lifecycle to use it in another intermediate function in my workflow. Should I do this via :onyx/params?

jeroenvandijk17:04:46

I thought i’ve seen an example, but I cannot find it now

lucasbradstreet17:04:45

If you’re generating a function, it’ll need to be injected into :onyx.core/params via a lifecycle function.

lucasbradstreet17:04:04

This is because :onyx/params in the task-map has to be serializeable

lucasbradstreet17:04:43

(defn inject-fn [{:keys [onyx.core/params onyx.core/task-map] :as event} lifecycle]
    {:onyx.core/params (conj params (fn [x] myfn))})

lucasbradstreet17:04:58

Something like that

lucasbradstreet17:04:47

Using {:lifecycle/before-task-start inject-fn} as a lifecycle-calls that you will refer to

jeroenvandijk17:04:00

Great, that works, thanks!