2021-08-25 jackdaw | Clojure Slack Archive

jackdaw

Leigh Peacock 2021-08-25T11:18:37.028600Z

Hi all! So I'm very new to Clojure and also Jackdaw, but decided to have a play and reimplement one of the Scala apps at work using Clojure, but following the Serdes example I'm struggling on the very last step trying to send a message to the output topic. I've hit the following issue which I've noticed in the issues on Github too, but I can't find a way past it:

java.lang.IllegalArgumentException: No implementation of method: :clj->avro of protocol: #'jackdaw.serdes.avro/SchemaCoercion found for class: nil

Basically the app reads in messages with a string key and an avro serde value, does some transformations and then spits out a long key and an avro serde value, but can't work out how to provide the kafka common serialization long serde for the key serializer, I'm probably down the rabbit hole, but tried this to mimic the string-serde provided by Jackdaw:

(ns deposit-normaliser-clojure.serdes
  (:gen-class)
  (:import org.apache.kafka.common.serialization.Serdes))

(defn long-serde
  []
  (Serdes/Long))

and then reference that in the config for the serde-resolver:

(def +topic-metadata+
  ... Omitted...
   :output
   {:topic-name "promos-engine-outbound-avro"
    :partition-count 1
    :replication-factor 1
    :key-serde {:serde-keyword :deposit-normaliser-clojure.serdes/long-serde}
    :value-serde {:serde-keyword :jackdaw.serdes.avro.confluent/serde
                  :schema-filename "outbound.events.avsc"
                  :key? false}}})

Sorry for the long message, and it's highly possible I'm missing something completely obvious, any help would be appreciated 🙂

✅ 1

2021-08-25T15:44:26.028700Z

I think you can also just replace

:key-serde {:serde-keyword :deposit-normaliser-clojure.serdes/long-serde}

with

:key-serde deposit-normaliser-clojure.serdes/long-serde

2021-08-25T15:44:57.028900Z

But do you know where in your topology this error is happening? when it's trying to produce the Avro before sending to the output topic?

2021-08-25T15:49:12.029100Z

tho the exception indicates there might be a problem with the schema you have

Leigh Peacock 2021-08-25T15:54:56.029300Z

Hi there, thanks for the pointers, the more I look at it the more I think it's something related to the schema too, it's the last step it fails on, when it goes to forward the message to the outbound topic:

stream-client [deposit-normaliser-9a471287-99db-4157-aa1f-907ec6168d0f] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.  - 2021-08-25 12:02:32,601  48472 [deposit-normaliser-9a471287-99db-4157-aa1f-907ec6168d0f-StreamThread-1] ERROR o.a.k.s.KafkaStreams
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic promos-engine-outbound-avro for task 0_8 due to:
java.lang.IllegalArgumentException: No implementation of method: :clj->avro of protocol: #'jackdaw.serdes.avro/SchemaCoercion found for class: nil
... Omitted...
Caused by: java.lang.IllegalArgumentException: No implementation of method: :clj->avro of protocol: #'jackdaw.serdes.avro/SchemaCoercion found for class: nil
	at clojure.core$_cache_protocol_fn.invokeStatic(core_deftype.clj:583)
	at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:575)
	at jackdaw.serdes.avro$eval2520$fn__2547$G__2511__2556.invoke(avro.clj:128)
	at jackdaw.serdes.avro.RecordType.clj__GT_avro(avro.clj:444)
	at jackdaw.serdes.avro.RecordType.clj__GT_avro(avro.clj:444)
	at jackdaw.serdes.avro$serializer$fn__2973.invoke(avro.clj:524)
	at jackdaw.serdes.fn_impl.FnSerializer.serialize(fn_impl.clj:25)
	at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
	at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:157)

2021-08-25T15:58:29.029500Z

weird. Maybe try and do a minimal Avro schema with 1 field and then just see if you can output that

Leigh Peacock 2021-08-25T15:58:36.029700Z

Waiiiiiiiit... Jackdaw mangles the names of the keywords doesn't it? So underscores become hyphens?

2021-08-25T15:58:43.029900Z

ah yeah lol

Leigh Peacock 2021-08-25T15:58:46.030100Z

Omg...

2021-08-25T15:58:53.030300Z

there's a mangle/unmangle functions

Leigh Peacock 2021-08-25T15:58:59.030500Z

It's been staring me in the face this whole time 😆

Leigh Peacock 2021-08-25T15:59:16.030700Z

Lemme try that

2021-08-25T16:00:20.030900Z

https://github.com/FundingCircle/jackdaw/blob/b4f67106cc12a78d9dcc64b6cba668ab6a804d04/src/jackdaw/serdes/avro.clj#L94-L98

(defn- ^String mangle [^String n]
  (str/replace n #"-" "_"))

(defn- ^String unmangle [^String n]
  (str/replace n #"_" "-"))

Leigh Peacock 2021-08-25T16:01:42.031300Z

Does it only do that for deserialisation or does this happen on serialisation too?

Leigh Peacock 2021-08-25T16:06:59.031500Z

I love you 😆

Leigh Peacock 2021-08-25T16:07:50.031600Z

I've spent days trying to work that out haha, sometimes just need a pointer in the right direction, sorry for being a muppet 😆

2021-08-25T16:11:13.031800Z

no worries, I have found jackdaw/clojure errors and exceptions can sometimes be cryptic

2021-08-25T16:11:28.032Z

> Does it only do that for deserialisation or does this happen on serialisation too? it happens on both directions

2021-08-25T16:12:55.032300Z

someone added a PR to make mangling optional over 2 years ago 😅 https://github.com/FundingCircle/jackdaw/pull/126

Leigh Peacock 2021-08-25T16:17:30.032600Z

Yeah, worst thing is I saw this and it didn't register it'd happen on serialisation too, feel like a right donkey haha

Leigh Peacock 2021-08-25T16:17:59.032800Z

They mention in the comments just need to hyphenate keywords facepalm

Clojurians Log v2

jackdaw