Fork me on GitHub
#jackdaw
<
2019-06-18
>
billh08:06:45

Hi guys, wondering if there is a way to completely shut off the avro schema coercion on deserialisation? I've run into a bug whereby I have an avro schema with a number of union types, I can see that the avro-record is correctly deserialised in jackdaw but then it looks like it goes through this coercion step but in doing so jackdaw seems to be selecting the wrong union record and the result is a deserialized message where the incorrect fields are entered with nil values - any fields that match in both unions are correctly displayed. I'll try and boil it down to an exact example and open an issue but in the meantime, I was wondering if I can turn this stage off completely? I tried to pass a nil type registry but it seems to still go through this step.

cddr08:06:32

Argh sorry about that. Must be super annoying. I don't know off hand but I'll have a look.

cddr09:06:13

@billh Here's an untested__ PR that might do what you need. Once it has finished running the tests it should publish a snapshot version to clojars. https://github.com/FundingCircle/jackdaw/pull/148

billh09:06:37

Thanks I'll give it a go and also I'll raise an issue once I boil my schema down to something more succinct

cddr09:06:57

Here's the maven co-ordinates for the snapshot version: "0.6.6-optional_no_coercion-SNAPSHOT"

billh09:06:54

Thanks I'll give it a go now just trying to figure out where to put the switch as I was using the serde-resolver but that doesn't accept it.

billh21:06:36

I forked it myself in the end and changed the resolver, feel free to dump the branch but thanks for that. I think I've nearly found the bug, I've narrowed it down to a schema that looks like this with union records in

{:type "record",
 :name "someevents",
 :fields [{:name "event",
           :type [{:name "wibbleEvent",
                   :type "record",
                   :fields [{:name "wibble", :type "string"}]}
                  {:name "wobbleEvent",
                   :type "record",
                   :fields [{:name "wobble", :type "string"}]}]}]}
If I setup a stream topology with the test driver etc and do a round trip publishing the following:
{:event {:wibble "wibble"}}
{:event {:wobble "wobble"}}
I get this outputted
{:event {:wibble "wibble"}
{:event {:wibble ""}
But the intermediate avro-data is correclty deserialised. Also oddly if I plug this into your unit tests (in jackdaw.serdes.avro-test) where they call (clj->avro ..) function directly you do get the correct output, it's only putting it through the full jackdaw workings I get the issue. I'll continue at it tomorrow, see if I can pinpoint it.