Fork me on GitHub
#jackdaw
<
2021-10-04
>
dakra15:10:46

@cddr I want to switch from the jackdaw included avro serde to yours https://github.com/cddr/edn-avro as I had errors with nested lists/maps in jackdaw. But with union types I always have to specify the union-path when I want to convert from edn to avro. So I have {:yolo "yolo"} but need {:yolo {:string "yolo"}} for the example in your code. How do you do it? Always "manually" or do you have some code/helper that you can share. I validate the output with malli so I'm thinking of writing a custom malli transformer but wondering how you handle it and what's the simplest way.

cddr15:10:23

Fortunately I never had to migrate any code from one to the other and the new code I wrote that used edn-avro just used the latter explicitly but I can see how migrating existing code would be annoying

cddr15:10:11

Also, you should probably know I don't actually use this anywhere in production. It was more of an itch I wanted to scratch but haven't yet had the opportunity to use it in a production system.

dakra15:10:59

Hmm, OK. Thanks. I want to write a relatively simple kstreams app that takes messages from one topic and splits them up (depending on some json field) to X other topics. Input is only json, output is avro as I want the data with the s3-connector to end up as parquet in S3. For some avro schemas with nested map in a list in map, jackdaw errors and I was looking for a simpler way than fixing jackdaw and reading your readme made me think that it's maybe better to not use it at all.

cddr15:10:02

I think you should be able to do that with jackdaw as it is currently no?

cddr15:10:41

Oh sorry I get it.

cddr15:10:23

But yeah the bug you've run into is exactly the reason I wrote edn-avro.

dakra15:10:00

I simply get

java.lang.IllegalArgumentException: No implementation of method: :clj->avro of protocol: #'jackdaw.serdes.avro/SchemaCoercion found for class: nil

dakra15:10:11

oh. thanks. Since I have malli schemas for it, shouldn't be hard to transform the output to the json format avro needs. I'll try that.

cddr15:10:23

Would be interesting to see what you come up with if you're permitted to share it. I'll happily add a docs section to the edn-avro project to document stuff like this because that's what people want to see 🙂 "ok you're project's great but how do I use it with...."

dakra16:10:30

Sure. I'll message you/open a PR if I have something useful.