Fork me on GitHub
#off-topic
<
2023-12-07
>
Stef Coetzee07:12:07

How does Nubank and others with large codebases and teams go about specifying interfaces between services/teams? In the comments of the HN entry on Jepsen being written in Clojure, there are some complaints around large teams and large codebases. E.g.: https://news.ycombinator.com/item?id=38545869 I'd love to learn more.

3
lispyclouds08:12:52

Not a large code base but for a big one like for https://github.com/bob-cd/bob I found the following approach work for inter-service interfaces and also have used it at times at work: • Define a "common" lib with specs/malli schemas: https://github.com/bob-cd/bob/blob/main/common/src/common/schemas.clj • These spec out the the messages that are sent across things like DB, queues (rendezvous points) etc • Each of the services depend on this common lib and when reading a message from a rendezvous point, it refers the spec and asserts correctness and does not when it writes to it • Each of services also uses these specs in its tests to assert the thing written to these points are correct by its varois functions; more to tests its own contract I found this approach work out quite well and scale too and having it shared makes it easier to maintain too. This is something I came up with so also keen to hear about others! 😄

gratitude 1
lispyclouds08:12:44

Oh yeah, also helps in generative tests too and negates the need for expensive end to end tests

Marcel Krcah13:12:40

I'm wondering about this as well, specifically about the following two things in the context of microservices. (Sorry for bit of a fuzzy and lengthy question; I'm not quite clear on this topic myself, plus the context I'm experiencing this is a large scala monorepo; not a clojure one) How is breakage approached? I can see at least two approaches: 1. Introduce a breaking change in the API or in the message format. However, that might require simultaneous update of dependent microservices, which complects the microservices and seems to go again the whole idea of independently deployable microservices. 2. Instead of breakage, accrete/grow as Rich described in Speculation; and perhaps have an internal channel to communicate changes and let other microservices/teams catch up when it's convenient for them. In a simple form, maybe sth like a spreadsheet with changes where teams express that they have migrated their clients/consumers to the latest version; and so when all teams migrate (in their own pace), the old/obsolete functionality could be safely removed. If the contract boundaries are captured in spec/schema/etc, approach one seem to imply changing the shared spec in a breaking way; approach two seem to imply growing the spec. In seems to me the approach two is the way to go; but it has a downside of services being ready to handle accretion. How are the contract boundaries technically shared across microservices? • If all microservices live in one git monorepo, each microservice might perhaps reach out to the common specs that lives in a shared directory. • If microservices live in separate git repos, maybe something like a shared lib or a git submodule might be needed? • Intuitively, I cannot stop thinking about implications of having one global lib/dir for all microservice specs. Is that a form of complexity? If microservice A doesn't consume microservice B; why should A care about updates of specs caused by B? • Another technical approach might be for each microservice to somehow publish their specs and let other microservices consume the specs of other microservices as they need? Any thoughts? I'd love to hear your experiences with such problems.

Stef Coetzee17:12:56

@U0609QE83SS, the Polylith folks do a great job of arguing against libraries, but if the overall system is in a fragmented state when you receive it, that might be an option. Option 2 brings to mind the strangler fig approach to change, popularized by Fowler. Much more sane than a sudden breaking change, I'm sure.