Fork me on GitHub
#onyx
<
2017-03-03
>
jasonbell11:03:20

Here's something that's been on my mind. Is there a confirmed way of knowing that a Kafka input stream task will always be "on" and attached to a peer. If so, how to check?

jasonbell11:03:08

Basically I want my job to run, forever without fear of peers backing off because they think there's nothing to do.

jasonbell11:03:55

Actually any tasks between the kafka input and any output for that matter, nothing can release back to the peer pool.

michaeldrogalis16:03:04

@jasonbell A job’s tasks are either all started or none. Nothing in between.

michaeldrogalis16:03:55

They’re started asynchronous because they’re running on different machines, but the decision to launch is consistent across them.

otfrom16:03:42

michaeldrogalis and the tasks remain started until the jvm process is killed?

michaeldrogalis16:03:24

@otfrom Or the job completes, or the scheduler reallocates those peers to a different job depending on what scheduler type you’re using.

jasonbell16:03:49

Okay, so with a kafka input stream on a balanced scheduler I'd expect the job to stay alive indefinitely.

jasonbell16:03:27

One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in /consumers node in zk.

jasonbell16:03:36

Is there somewhere else it's stored.

otfrom16:03:37

jasonbell this might help with figuring out Aeron shared memory stuff https://github.com/real-logic/Aeron

jasonbell17:03:31

@michaeldrogalis <<One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in /consumers node in zk.>> Can you help me with this one please.

michaeldrogalis17:03:13

Kafka offsets go along with the task checkpoint.

michaeldrogalis17:03:25

Is your peer writing checkpoints to ZooKeeper or S3?

michaeldrogalis17:03:00

Should be under /onyx/<tenancy id>/checkpoint AFAIK. I might be wrong on that one though, I didnt upgrade the Kafka plugin

jasonbell17:03:27

I'm wondering if that makes it difficult to users to tally the Kafka offset with where the Onyx check point is. I can't run ConsumerGroupCommand for example as it won't find the consumer group in the /consumers node in kafka. I have to refer to the Onyx one only.

lucasbradstreet18:03:59

@jasonbell the offsets are stored in the s3 checkpoints now

jasonbell18:03:45

I would like to get to the stage of being able to safely say "This number of messages are in the topic, this many Onyx processed"

jasonbell18:03:24

I see my zk node /onyx/[id]/checkpoint/[jobid]/ and then some nice numbers.

lucasbradstreet18:03:06

It depends on whether you're using the ZooKeeper checkpoint implementation or the s3 one. I believe those nodes were written by 0.9?

lucasbradstreet18:03:55

You want to do this for monitoring sake?

jasonbell18:03:10

Well for sanity sake... 🙂

lucasbradstreet18:03:49

Understood. I'm just wondering if it would be enough for the plugin to report a metric

jasonbell18:03:12

that would be good

jasonbell18:03:21

[org.onyxplatform/lib-onyx "0.10.0.0"]
                 [org.onyxplatform/onyx-kafka "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-amazon-s3 "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-metrics "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-peer-http-query "0.10.0.0-beta5"]

jasonbell18:03:45

I cleared ZK onyx nodes out before redeploying and they're back there again so I don't think it's a 0.9 thing.

jasonbell18:03:56

It's Kafka 10, Onyx 0.10

jasonbell18:03:11

More later, I need to go here for a bit.

lucasbradstreet18:03:22

Ok thanks. I'll have a look at that one then. I thought we were scoping it differently but now I remember that it's only the coordinates that we're not scoping on the tenancy id

lucasbradstreet18:03:38

Sure. I'll have a think about what we can do to make this easier. It's definitely a good thing

jasonbell18:03:52

If you think of anything else you feel I need to know then please drop them here (or PM) and then I'll pick them up later.

jasonbell18:03:12

@lucasbradstreet yes, thanks, I think a metric from the plugin would be a great idea,