at the repl I will get a handful of "real" responses and put those into my tests as mocks, and then either redef or write a test impl of the relevant Component to return the mock. If appropriate, I sometimes also reach for test.check to help generate ranges of inputs for my tests

I am working with wiremock at the moment... don't know if it is good in the long run

Wiremock is cool, we use that in some cases. It's a little cumbersome for us, but I can see how it would be helpful.

I think a little more info would help you get a good answer here. Does this app have any pure logic apart from these service interactions? What kind of interactions, and how widespread? How large is the service’s interface?

yeah, so the external service is odoo and it's got a JSON RPC API and we don't have a lot of pure logic in the webapp... it's more of a user-interface. We do a lot of back and forth of api-requests to transform stuff. we read more than we write.

So essentially your app transforms data from the service into some other shape?

right

How unstable/complicated are the other service’s responses? If you just saved a few of them would you feel comfortable with your coverage, or are you worried they’d throw you a curveball?

I don't have enough experience yet... I assume there are some edge cases that break things (like new plugins installed by some other person)... the thing is odoo with all it's stuff is nearly impossible to set up for a testing env from scratch... so we would have to use an instance that is long running (and most probably in flux)

Yeah if you have any doubt as to the stability of the response shapes then I’d go for a live/wiremock approach, maybe with some hard coded unit tests on any particularly complex transformations

Especially since it’s read only there’s not much risk to testing live

If test runtimes are a problem, maybe snapshot responses 1/day or something

I’d probably weight my test distribution heavily towards e2e tests too since if something breaks, the only way you might be able to tell is if something goes wrong in the frontend. You know more about your situation than I do of course, this might or might not be good advice, depending on your circumstances

sure thanks, that's the way I am currently going... will try mocking it but don't know if it is feasible on the long run.

New library in this space that might be interesting: https://github.com/griffinbank/test.contract

My go-to testing strategy for these kinds of circumstances is vcr testing (inspired by the ruby gem https://github.com/vcr/vcr)

Basically your tests can run in two different modes.

First, you can run them in "live" mode, where real requests to the external service are made. This allows you to iterate until you have a real request that works with the service. You capture the response (as a .json file)

Second, you can run the same tests in "replay" mode. Here the http client looks for a json file that fits the request (e.g. by hashing the path/method/body). If such a file is found, it replays the interaction with the server as seen before.

In this second mode, the test runs as fast as any microtest (unit test)

At this point you can keep refactoring and use the test based on the recorded interaction as a regression test

Technically, a spy server like wiremock should give you a bit more confidence because you talk to a real http server, but I feel vcr-testing is in many cases good enough.

I've used this to build clients for the circleci api and the like

VCR is simply amazing. It's not limited only for HTTP requests, I once mocked a whole database with VCR :)

did you use vcr-clj?

Yep, the CLJ version. Probably the best version of VCR I ever used, honestly

I ended up writing my own (just to have full control) but vcr-clj looks great as well

This question is really the elephant in the room when it comes to testing modern applications, because for a lot of apps most of what they do is glueing together third party services. It really changes how to think about testing.

I liked the approach of vcr but hato does not work well with it. Serialization is a problem with vcr.