This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # admin-announcements (1)
- # aws (2)
- # beginners (21)
- # boot (28)
- # braid-chat (1)
- # cider (51)
- # cljs-edn (7)
- # cljsjs (35)
- # cljsrn (2)
- # clojure (85)
- # clojure-chicago (7)
- # clojure-czech (1)
- # clojure-gamedev (3)
- # clojure-poland (2)
- # clojure-russia (80)
- # clojure-sanfrancisco (1)
- # clojure-uk (5)
- # clojurebridge (9)
- # clojurescript (68)
- # cursive (29)
- # datomic (23)
- # emacs (2)
- # hoplon (94)
- # jobs-discuss (15)
- # juxt (2)
- # liberator (2)
- # luminus (16)
- # mount (12)
- # off-topic (7)
- # om (57)
- # onyx (58)
- # proton (10)
- # re-frame (9)
- # reagent (38)
- # remote-jobs (2)
- # rum (12)
- # untangled (136)
We are using an older version of Kafka in production than the one onyx-kafka uses. I think there are subtle differences in how Kafka topics are registered in Zookeeper. So far it looks like the new client cannot read from topics generated with an old client (still investigating). Would it be an option to make a fork of onyx-kafka and downgrade the kafka dependency? Or is onyx-kafka using specific features of kafka (clients)?
I saw that the zookeeper dependency is the same for the different libraries so that should be fine
e.g. Pin [apache.kafka/kafka_2.10 "0.8.2.1"] [org.apache.kafka/kafka-clients "0.8.2.1"]
When starting a new job (single peer) that writes to a kafka topic, where the kafka hasn’t had any topic configured. I consistently loose one or more batches (batch size is 10). This is fixed by ensuring that the topic is created before submitting the job (my input is a seq with 10.000 uniq ids) Nothing in the onyx logs to indicate any errors.
I wonder if it had to do with https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-1006
Are you immediately trying to read from the topic? If so, can you add a sleep before you try to take the segments to help diagnose whether the problem is on the consumer?
After my docker-compose up ( with zookeeper, kafka and a single peer) I attach to the kafka container and runs the console consumer. Afterwards I run the submit job. I can very quickly push my repo up to github
If I consumes after the submit and the job is finished (the job is very quickly finished), the topic is empty. If I consumes during the submit I loses some messages.
The fact that the topic is empty if you start the consumer later is pretty odd
That to me indicates that there's something odd going on in the consumer e.g. it's not reading from the start of the dream
That fixed it. So the issue was not with onyx, but the way I used the console consumer.
No problem. After you said that it read nothing if you started it later I was pretty sure it wasn't onyx :)
Speaking of Kafka, is anyone on Kafka 0.9 yet? I've upgraded onyx-kafka for 0.9 on another branch.
Need to figure out a way to have both versions of the plugin for 0.8 and 0.9 side-by-side since 0.8 is still being used a lot.
Do people use docker-compose configuration provided by onyx-template to develop/test kafka related jobs? I find it impossible to customize the Zookeeper config (for number of max clients). I’m thinking of building my own image for zookeeper. I’m new to docker-compose and I find that part particularly pleasant
so just one container, but i have too many peers and I also have an external zookeeper client. The funny thing is that i only have this problem while reading
Will continue tomorrow. I think i’ll just customize the docker container and try again
Oh, yeah @jeroenvandijk. I hit this problem with stock Docker images all the time 😕
Customizing the config can get messy, especially if the author did that weird Sed trick that's so common
@ckarlsen: Cool, thanks! Still needs to undergo a lot of testing, but it's worth a shot.
I still don't understand why the kafka plugin has to write :done to the topic? Will cause problems on compacted topics (requires a key)
@ckarlsen: From the output writer? Yeah we've been meaning to make that configurable to turn off.
It was just an idea I had from a long time ago. Works alright in some circumstances when you want a sentinel value, but being able to turn it off would be good too
I had to remove it. Caused me some headaches when I tried to run some batch jobs for the first time
If you want to send over a PR that accepts
:kafka/no-seal? in the map and conditionally doesn't write
:done, I'd be happy to merge it
Otherwise make a ticket. One of those things I want to do, but I dont have 10 minutes free these days. Heh