This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-06-29
Channels
- # announcements (10)
- # babashka (18)
- # beginners (136)
- # calva (9)
- # cider (14)
- # clara (12)
- # clj-kondo (32)
- # cljsrn (3)
- # clojure (133)
- # clojure-europe (21)
- # clojure-nl (4)
- # clojure-uk (15)
- # clojurescript (60)
- # conjure (40)
- # cursive (12)
- # datomic (6)
- # emacs (2)
- # fulcro (19)
- # jackdaw (25)
- # jobs-discuss (3)
- # kaocha (3)
- # leiningen (5)
- # off-topic (99)
- # pedestal (1)
- # re-frame (49)
- # reagent (4)
- # ring (5)
- # rum (5)
- # shadow-cljs (53)
- # spacemacs (2)
- # sql (13)
- # timbre (2)
- # tools-deps (23)
- # vim (11)
- # xtdb (7)
Hey @dominicm, sorry I missed this last week. Do you mean this get-records
? https://github.com/FundingCircle/jackdaw/blob/9b5bbd0d4582213ee43eb6af3ea37dfd6129d82a/dev/jackdaw/repl.clj#L70. Did you figure it out?
https://bin.disroot.org/?6b122f22678a1bac#HJgQujM4wuvNuttPQx6TLjKJAB2F7byEkwWwEr9uBUvS
Hm, that actually worked for me. To provide the kafka env, I used the docker-compose setup described here https://grumpyhacker.com/test-machine-test-env/, and I had to adjust the bootstrap server setting to be "localhost:19092" instead of "localhost:9092", but once I did that, and after evaluating the line that publishes the line of text, when I evaluated get-records
, I got the records that had been published (4 records, one for each unique word).
I connected to an in-memory one that I ran inside the JVM. Docker doesn't work for me. Do you think the problem is likely to be in the kafka server?
I think running it embedded should work ok. Is there one of those bundled with crux? Is that what you're using?
io.dominic.jackdaw=> (publish (:input topic-metadata) "hello there kafka streams 2")
nil
io.dominic.jackdaw=> (get-records (:input topic-metadata))
()
io.dominic.jackdaw=>
This behavior confuses me.Hm, when I used the embedded server from crux (https://github.com/juxt/crux/blob/master/crux-kafka-embedded/src/crux/kafka/embedded.clj#L86), I can't even create the topics. Seems to just hang.
Crux definitely uses this and works, and this is exactly what the cli calls into. Hmm.
Yeah I'm sure it does. I think it must be something weird. Anyway, I'll play around for a bit and see if I can get it working with the embedded server. There's probably some logging somewhere indicating why it's not working. This is why I think I ended up preferring docker-compose. It seemed easier to find the logs and distinguish them from my own app logs.
Ah, so with logging enabled, I think I see what the problem is (at least for me not being able to create topics)....
2020-06-29 14:23:27 ERROR KafkaApis - [KafkaApi-0] Error when handling request: clientId=adminclient-1, correlationId=18, api=METADATA, body={topics=[],allow_auto_topic_creation=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false}
java.lang.NoSuchMethodError: 'org.apache.kafka.common.requests.MetadataResponse org.apache.kafka.common.requests.MetadataResponse.prepareResponse(int, java.util.List, java.lang.String, int, java.util.List, int)'
Seems like there's a version conflict between the kafka stack used by the embedded version of kafka, and that being used by jackdaw.
So I was able to reproduce the problem of get-records
returning empty by adding "kafka-clients" to the exclusions list when depending on jackdaw. However, for me, the data really did exist on the output topic which could be confirmed by using the standard kafka-console-consumer command line client to read the topic from the beginning.
I'm using kafka 2.3.0 of kafka-clients, streams, streams-test-utils and kafka_2.12. So those should all be OK, no?
{:deps
{fundingcircle/jackdaw {:mvn/version "0.7.4"}
org.clojure/clojure {:mvn/version "1.10.1"}
org.apache.kafka/kafka-streams {:mvn/version "2.3.0"}
org.apache.kafka/kafka-streams-test-utils {:mvn/version "2.3.0"}
org.apache.kafka/kafka-clients {:mvn/version "2.3.0"}
org.apache.kafka/kafka_2.12 {:mvn/version "2.3.0"}
org.slf4j/slf4j-simple {:mvn/version "1.7.30"}
org.apache.zookeeper/zookeeper {:mvn/version "3.5.7"
:exclusions [io.netty/netty jline org.apache.yetus/audience-annotations org.slf4j/slf4j-log4j12 log4j]}
org.clojure/data.csv {:mvn/version "1.0.0"}}
:mvn/repos
{"confluent" {:url " "}}}
my deps.edn ftr.