Fork me on GitHub
Ben Sless06:05:28

Conversation I had with QA at work, curious to hear your thoughts - is it reasonable to negative tests to check for specific error messages? I'm using malli and reitit for a web server, and while I feel free to commit to the error message and the structure of details, I wouldn't want to commit to the values for the keys under details

Ferdinand Beyer11:05:19

I think this is hard to answer. My first instinct would be to test mainly for semantics, and leave details that might change without breaking the logic out of it. E.g. for error messages, I’d try to get a machine readable error code / number and check for that. Important to make sure I get a “404” when something does not exist, but the test should not fail when we decide to change the human-readable description. However, there could be cases where the “message” directed to a human audience is considered business critical. Maybe in legal contexts? In this case, it seems reasonable to test for exactly that.

Ben Sless11:05:17

I was willing to commit to more, which was status code, text message, and details structure. Just not the fine details. This is also not precisely a general public facing API, but for our SDK


we typically only commit to structural validation + value types, not actual values WRT error messages.


Some things, like error codes/type we do validate because those are more fixed and stable.