This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # bangalore-clj (2)
- # beginners (29)
- # boot (52)
- # cider (4)
- # clara (3)
- # cljs-dev (34)
- # cljsjs (7)
- # cljsrn (3)
- # clojure (71)
- # clojure-austin (1)
- # clojure-dev (5)
- # clojure-france (20)
- # clojure-russia (51)
- # clojure-spec (9)
- # clojure-uk (20)
- # clojurescript (131)
- # core-async (56)
- # core-logic (6)
- # cursive (50)
- # datascript (19)
- # datomic (16)
- # dirac (118)
- # emacs (100)
- # events (4)
- # hoplon (14)
- # incanter (1)
- # jobs (7)
- # jobs-discuss (96)
- # jobs-rus (21)
- # lein-figwheel (5)
- # leiningen (21)
- # off-topic (11)
- # om (45)
- # onyx (42)
- # pamela (1)
- # pedestal (22)
- # portland-or (3)
- # re-frame (8)
- # reagent (5)
- # ring (9)
- # robots (1)
- # spacemacs (14)
- # specter (28)
- # sql (2)
- # untangled (165)
Here's something that's been on my mind. Is there a confirmed way of knowing that a Kafka input stream task will always be "on" and attached to a peer. If so, how to check?
Basically I want my job to run, forever without fear of peers backing off because they think there's nothing to do.
Actually any tasks between the kafka input and any output for that matter, nothing can release back to the peer pool.
They’re started asynchronous because they’re running on different machines, but the decision to launch is consistent across them.
@otfrom Or the job completes, or the scheduler reallocates those peers to a different job depending on what scheduler type you’re using.
Okay, so with a kafka input stream on a balanced scheduler I'd expect the job to stay alive indefinitely.
One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in
/consumers node in zk.
jasonbell this might help with figuring out Aeron shared memory stuff https://github.com/real-logic/Aeron
@michaeldrogalis <<One thing I've noticed is that I don't see where the consumer is written to in Zookeeper, I don't see anything in
/consumers node in zk.>> Can you help me with this one please.
Should be under /onyx/<tenancy id>/checkpoint AFAIK. I might be wrong on that one though, I didnt upgrade the Kafka plugin
I'm wondering if that makes it difficult to users to tally the Kafka offset with where the Onyx check point is. I can't run
ConsumerGroupCommand for example as it won't find the consumer group in the
/consumers node in kafka. I have to refer to the Onyx one only.
I would like to get to the stage of being able to safely say "This number of messages are in the topic, this many Onyx processed"
It depends on whether you're using the ZooKeeper checkpoint implementation or the s3 one. I believe those nodes were written by 0.9?
Understood. I'm just wondering if it would be enough for the plugin to report a metric
[org.onyxplatform/lib-onyx "0.10.0.0"] [org.onyxplatform/onyx-kafka "0.10.0.0-beta5"] [org.onyxplatform/onyx-amazon-s3 "0.10.0.0-beta5"] [org.onyxplatform/onyx-metrics "0.10.0.0-beta5"] [org.onyxplatform/onyx-peer-http-query "0.10.0.0-beta5"]
I cleared ZK onyx nodes out before redeploying and they're back there again so I don't think it's a 0.9 thing.
Ok thanks. I'll have a look at that one then. I thought we were scoping it differently but now I remember that it's only the coordinates that we're not scoping on the tenancy id
Sure. I'll have a think about what we can do to make this easier. It's definitely a good thing
If you think of anything else you feel I need to know then please drop them here (or PM) and then I'll pick them up later.
@lucasbradstreet yes, thanks, I think a metric from the plugin would be a great idea,