Clojurians Log v2
jackdaw
Hello 👋
Is there a complete example anywhere for using jackdaw.serdes.avro.confluent/serde?
I'm pretty sure the schema-registry-url param is just the base URL for the registry.
The schema param I am less sure about. I can sort of see in the tests that it's the actual Avro JSON schema.
Where is this supposed to come from? Do I have to have schemas locally and in the schema registry? Is there a way to retrieve the schema from the registry with Jackdaw or is this something I'm responsible for?
And one other question: How are references supposed to work? If I try to initialize a serde with a local schema that doesn't use a reference it's all good, but if I try one that uses a reference, I get a parse error.
Producer will need to specify both. schema is the writer schema for encoding, schema-registry-url is the registry for the producer to lookup schema id by writer schema (if the writer scheme doesn't exist or is updated, a new id will be assigned). Scheme id is encoded in the message by writer/producer so reader would know which scheme (in the scheme registry) to use to decode message.
I haven't figured out how to explictly provide an Avro schema to jackdaw, IIRC I thought I had to go through the schema-registry, which I have not yet enabled/configured for my cluster, so I had to resort to other Java-interop-unnatural-acts to make it work.... If you have any pointers/advice about how I could directly provide an Avro schema to a Jackdaw serde, I would be most grateful
@dcj as far as I can tell the serde function accepts a parameter called schema if you pass it a string that contains the JSON representation of the Avro schema, it'll load it just fine.
https://github.com/FundingCircle/jackdaw/blob/master/src/jackdaw/serdes/avro/confluent.clj#L11
Is that what you mean? In this case you could read the file in from resources and pass the string to the serde function.
Whether or not this is how it should be done, I'm not entirely sure.
Thank you, I will give that a try....
So, if I am passing in a JSON string as schema then schema-registry-url and key? can just be nil?
Well, I realized after that I was just telling you about the confluent serde and you had said you aren't using schema registry yet. So I'm not sure that'll work for you. Wish I had more info for you, but I'm just getting into it myself and docs are sparse.