Fork me on GitHub
#jackdaw
<
2020-09-22
>
Raziyeh Mohajer07:09:14

Hello everyone, I want to use a key.serializer and value.serializer for kafka producer other than Kafka own serializers with jackdaw. this is my code for producer

(def serialize nippy/freeze)

(def Kafka-Edn-Serializer
  (reify Serializer
    (serialize [this topic headers data]
      (serialize data))
    (serialize [this topic data]
      (nippy/freeze data))
    (configure [this _ _])
    (close [this])))

(def producer-config
  {"bootstrap.servers" "localhost:9092"
   "key.serializer"    (-> Kafka-Edn-Serializer
                           .getClass
                           .getName)
   "value.serializer"  (-> Kafka-Edn-Serializer
                           .getClass
                           .getName)
   "acks"              "all"
   "client.id"         "foo"})

(defn -main []
  (jc/producer producer-config)
  )
I can run it in repl with spacesmacs editor but when I use lein run to run this I got following error
Invalid value producer_example$reify__12915 for configuration key.serializer: Class producer_example$reify__12915 could not be found.
How should I solve this problem?

Daniel Stephens14:09:10

pretty sure you can give it as an instance instead of as the class name which might fix your issue @raziyehmohajer.rms did you try:

{"bootstrap.servers" "localhost:9092"
 "key.serializer" Kafka-Edn-Serializer
 "value.serializer" Kafka-Edn-Serializer
 "acks" "all"
 "client.id" "foo"}

Raziyeh Mohajer17:09:36

@ULNRSUK8C yes I tried that too I got this error

Invalid value producer_example$reify__38516@76bd8d56 for configuration
   key.serializer: Expected a Class instance or class name.

Daniel Stephens08:09:13

Ahh seems my use is slightly different:

(client/producer
     {:bootstrap.servers brokers
      :client.id client-id
      :acks "all"}
     {:key-serde mj/key-serde
      :value-serde mj/value-serde})
Note that I provide two args instead of one, looks like the implementation in this case passes these directly in as instances.

Daniel Stephens08:09:10

These are expected to be serdes but hopefully that's okay, they only have the .serializer side called in honesty

Daniel Stephens15:09:41

is jackdaw still alive 😬 I've been hoping https://github.com/FundingCircle/jackdaw/pull/270 would get merged for quite a while

Darin Douglass15:09:24

last i heard (months ago) covid hit the admin's schedule hard or something. iirc it was @noisesmith who said that

noisesmith15:09:37

it was work related to covid that displaced jackdaw work, I (along with the rest of the former US eng team for fc) am no longer working with fc so can't tell you the current situation

Darin Douglass15:09:30

thanks for the update, it was what i remembered. sorry about the loss of a job. hopefully you found something quick enough

Daniel Stephens15:09:43

Thanks, both, for the information 👍

dakra20:09:34

So does that mean that most (all?) people that developed jackdaw don't have write access to the repo anymore to maintain it? If so, then maybe it would even make sense to fork it and put the original authors back as maintainers? I like jackdaw and currently use it for my project. I, too, would like an update to a newer Kafka version and also very interested in the transducer PR.

👍 6
markt09:09:28

That sounds like a great idea, keen for not too many forks to be made for this.

Daniel Stephens09:09:31

Agree, I guess it might be unfair to expect all the original maintainers to come along, but considering the number of PRs it seems like there would be plenty of other people willing to help, myself included!

dakra12:09:30

The thing is that the last commit was only 15. July. So if they're just busy a bit that's perfectly acceptable imho. But if it's really the case that quite a few people would like to contribute but can't because it's still under the fc company account where nobody has access to, then it's different. Personally I just see a bunch of PRs that look good and which I would like to see merged.

noisesmith19:09:21

I think you still misunderstand - dev moved to london, there's still an active eng team, that team might not be as invested in jackdaw as the us team was but the company still has clojure devs, and last I heard was hiring, just consolidating in london

👍 6
noisesmith19:09:37

I can't really speak on the rest

Daniel Stephens19:09:58

ahh thanks @noisesmith I had indeed misunderstood. That's more promising, might just need to give it some time after the upheaval to get going again 👍

dakra07:09:52

ok. thanks for the clarification @noisesmith. Hope they pick up development of jackdaw soon. Quite a few PRs seem ready and just waiting for the merge button to be pressed 😉

Darin Douglass15:09:49

i too am interested in a couple of the pr's currently open against the lib