Here's something that's been on my mind. Is there a confirmed way of knowing that a Kafka input stream task will always be "on" and attached to a peer. If so, how to check?

Basically I want my job to run, forever without fear of peers backing off because they think there's nothing to do.

Actually any tasks between the kafka input and any output for that matter, nothing can release back to the peer pool.

@jasonbell A job’s tasks are either all started or none. Nothing in between.

They’re started asynchronous because they’re running on different machines, but the decision to launch is consistent across them.

ok, ta

michaeldrogalis and the tasks remain started until the jvm process is killed?

@otfrom Or the job completes, or the scheduler reallocates those peers to a different job depending on what scheduler type you’re using.

ok.

Okay, so with a kafka input stream on a balanced scheduler I'd expect the job to stay alive indefinitely.

One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in /consumers node in zk.

Is there somewhere else it's stored.

jasonbell this might help with figuring out Aeron shared memory stuff https://github.com/real-logic/Aeron

thanks

@michaeldrogalis <<One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in /consumers node in zk.>> Can you help me with this one please.

@jasonbell Which Onyx version?

0.10

Kafka offsets go along with the task checkpoint.

Is your peer writing checkpoints to ZooKeeper or S3?

zk

Should be under /onyx/<tenancy id>/checkpoint AFAIK. I might be wrong on that one though, I didnt upgrade the Kafka plugin

thanks

I'm wondering if that makes it difficult to users to tally the Kafka offset with where the Onyx check point is. I can't run ConsumerGroupCommand for example as it won't find the consumer group in the /consumers node in kafka. I have to refer to the Onyx one only.

@jasonbell the offsets are stored in the s3 checkpoints now

by default?

I would like to get to the stage of being able to safely say "This number of messages are in the topic, this many Onyx processed"

I see my zk node /onyx/[id]/checkpoint/[jobid]/ and then some nice numbers.

It depends on whether you're using the ZooKeeper checkpoint implementation or the s3 one. I believe those nodes were written by 0.9?

You want to do this for monitoring sake?

Well for sanity sake... 🙂

Understood. I'm just wondering if it would be enough for the plugin to report a metric

that would be good

[org.onyxplatform/lib-onyx "0.10.0.0"]
                 [org.onyxplatform/onyx-kafka "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-amazon-s3 "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-metrics "0.10.0.0-beta5"]
                 [org.onyxplatform/onyx-peer-http-query "0.10.0.0-beta5"]

I cleared ZK onyx nodes out before redeploying and they're back there again so I don't think it's a 0.9 thing.

It's Kafka 10, Onyx 0.10

More later, I need to go here for a bit.

Ok thanks. I'll have a look at that one then. I thought we were scoping it differently but now I remember that it's only the coordinates that we're not scoping on the tenancy id

Sure. I'll have a think about what we can do to make this easier. It's definitely a good thing

If you think of anything else you feel I need to know then please drop them here (or PM) and then I'll pick them up later.

Will do

@lucasbradstreet yes, thanks, I think a metric from the plugin would be a great idea,

ttfn