Fork me on GitHub
#architecture
<
2022-07-07
>
Dave Russell19:07:58

Hey folks! Any tips on building modules in a monolithic Clojure webapp? We have a fairly large and growing monolithic codebase and it seems that separating core pieces into well-defined modules with good interfaces would give us a lot of value. A layered architecture has been great for us, especially when the team and codebase was smaller. As the team has grown, there has emerged a need to separate code function / ownership / and encapsulate functionality s.t. the other components in the system share a common, consistent interface. Do you guys have any useful guidelines / libraries / tooling etc on how to structure things into "modules"? Code organization, interface definition, etc. Anyone go through a similar process? Mostly looking for food for thought in how such things can be structured 🙂

jumar10:07:35

I do not have experience with it but #polylith sounds like something you want to explore 🙂

👀 1
Dave Russell11:07:52

Thanks! I've been diving in and I think it solves a lot of similar problems to what we have 🙂

Drew Verlee14:07:20

Can you give a concert example of something you would like to do but can't do? What do you mean by "code ownership" isn't that controlled through something like github? Clojure has many ways to encapsulate or scope logic. Defn is one.

Dave Russell20:07:18

Happy to! My problem isn't how to do encapsulation in Clojure the language; it's how best, in a mediumish (150K loc) Clojure monolith, with 20ish developers, to encapsulate pieces of functionality in order to enable productivity across the teams. We have a layered architecture now, but have seen issues where, for example, code in one part of the codebase reaches too deeply into another part of the codebase. This is what I mean by code ownership and encapsulation -- I want to architecture the codebase s.t. it's easy and productive for a team to build the features they're working on, and do it in such a way that 1) it minimizes friction for developers on other teams (I guess you could call this uniformity and encapsulation) 2) they don't accidentally reach too deeply into other teams' work (I guess you could call this interfaces). Obviously developer discipline, documentation, knowledge sharing helps with this, and obviously "build modular code, duh!" is an easy response -- but that's basically what I'm looking for: how do others in the Clojure community design modular monoliths.

Dave Russell20:07:17

I think Polylith has actually hit this pretty well on the head: a nice lightweight vocabulary that gives uniformity (both within the project/teams and between projects, if they're both Polylith apps), and proposes a directory structure that gives encapsulation and an interface. So I dig it 🙂

Rupert (All Street)21:07:02

A couple of thoughts: • Clojure is well suited to small libraries. ◦ They're fun to use and fun to write. • Dependency injection works well to make code more modular and remove fixed dependencies in code. ◦ At All Street we use https://github.com/weavejester/integrant for this. Here's an approach that worked well at All Street: • Keep the monolith (rather than microsevervices) • Extract out some libraries into new projects (either same git repo or use git submodules). • Use a multi project build tool like https://github.com/ruped/lein-multi-modules (a fork of lein modules with multithreading) or perhaps deps.edn. That way you build all your libraries with your existing monolith (no maven repo required). We've scaled this approach to over 100+ libraries with barely a hitch.

👍 1
Drew Verlee21:07:17

> code in one part of the codebase reaches too deeply into another part of the codebase I understand the feeling your projecting, but i think if you refine specifically why this is an issue it will help you identify the problem. I firmly believe this has to be tackled on a case by case basis. Identifying key abstractions that are useful to the business is important. Those should be split into libs and/or new shared services/microservices/databases/what-ever-is-appropriate. At the same time, if the business never harmonizes it's efforts, the code base has to follow suit. It's meaningless to put walls in a wild garden and call it progress.

Drew Verlee22:07:58

abstractly speaking, it's necessary for developers to do the thing that achieves success in the shortest possible step and time investment. They cannot afford to take bigger steps because the risk reward isn't in favor of it. Change the risk reward for larger steps in areas that have been identified by the teams as having issues and you might be surprised what solutions flow upwards.

Dave Russell22:07:43

Thanks for the responses! @UJVEQPAKS we use Component for dep injection and also intend to keep things monolithic in general. Sounds like you guys are addressing modularity with subprojects/libraries. Have you faced versioning issues for shared libs? Are your APIs separate projects as well, or do you have a main project with core components parceled out as sub projects?

Dave Russell22:07:22

@U0DJ4T5U1 > Those should be split into libs and/or new shared services/microservices/databases/what-ever-is-appropriate This is the heart of my question -- our code is already organized by key abstraction. The issue is that the splitting in the codebase today has lack of clear definition where the boundaries of the splits lie due to how things are organized, so I'm reaching for stronger separation. So the "what-ever-is-appropriate" is the answer to the question I'm asking 🙂

Dave Russell22:07:18

> Change the risk reward for larger steps in areas that have been identified by the teams as having issues and you might be surprised what solutions flow upwards Interesting ideas! I don't think it's wrong incentives or developers optimizing for small steps -- it's the lack of a codebase-wide uniform way of constructing building blocks that encourages modularity while also enabling each team to have agency

Rupert (All Street)12:07:05

> Have you faced versioning issues for shared libs? We take the mono repo approach to versioning: All code uses the same version of all libraries (typically the latest stable code that is available on master branch). Most library changes are backwards compatible; if the change is not backwards compatible then the library maintainer sends PR fixes to all downstream projects that they have access to. It means that you don't end up building an awesome new library versions that nobody is using. This happened many times at previous companies I worked at. > Are your APIs separate projects as well, or do you have a main project with core components parceled out as sub projects? Applications and libraries are their own standalone clojure project in separate git repos (but all built and tested together like a mono repo as described earlier). So we get fine grained permissioning of source code access using GitHub teams. We can also generate graphs of the dependency tree between all applications/libraries.

🙏 1
didibus00:07:41

@U0DJ4T5U1 I think you bring a good point. One thing as you seperate responsibilities between tech teams, you have to find ways so that projects and features minimize the amount of teams involved to deliver them. It's tempting to split by layers. One team owns the data storage, another the authentication, another the payments, etc. Now in truth, I've only got experience with this split. But I've been thinking a lot about a vertical split instead, where you split by lanes, aka, split by user features. If one feature needs auth, storage, payment, etc. And a team build it all but dedicated to that one feature. It seems you'd only need one team for every feature. If instead you've got your teams split, then all new features might be all teams to make changes if it requires changes to auth, storage, payments, etc. So in my opinion, but I've not been able to truly see this practice to know how well it would work, I think it's best to have teams building internal SaaS in a way, like a service than you can use to easily add authentication to your own app, another to easily add payment capabilities, another to more easily manage storage, etc. Where the priorities of the features each internal SaaS will have is based on your internal business needs. And then have other teams dedicated to owning chunks of the business features/use cases end to end, but it should now be that even though they're just one team, adding auth, storage, payment, etc. shouldn't be out of reach for them now that they can leverage the other team's SaaS for it all.

didibus00:07:46

So for example, instead of having one team own the auth for everything, which can easily become a bottleneck as the demand for auth from all the things become too many for one team to fit them all in the year, and now some projects are blocked because the auth team can't quickly enough make the necessary changes to their auth for all the active projects. Instead of that, you've got one team making it easier and lower effort required for every other team to own their own auth. So now, let's say there's 5 active projects, one for each of say 5 different team, and they all need something different in the auth requirements. Instead of all team going to the one auth team and saying, can you make those changes, and now they have to sequence them since they can't just make all of them in parallel, maybe only two at a time, if each takes 12 weeks, some projects are kind of blocked just waiting for bandwidth on the auth team. Instead, each of those team can just make the changes to auth they need, since they own their own auth. And ideally the auth team SaaS/libs should make that easy for each team.

didibus01:07:48

The other benefits of this is that if you end up with a really successful internal SaaS that's now a new business opportunity to externalize the SaaS and actually sell it.

niquola21:08:04

Take a look at our https://github.com/zen-lang/zen. I'm working on "zen-system", which should help to split the system into model-driven modules.

seancorfield19:07:21

Polylith? We're about a third of the way into a migration to that and I really like the focus on naming and decoupling components, as well as the separation of "apps" from reusable components from "projects" (building artifacts).

👍 2
polylith 3
Dave Russell19:07:42

Thank you! We recently also had a monorepo migration (for the non-clojure parts) so I'm looking forward to reading through this 🙂

seancorfield19:07:27

Feel free to DM me with any Qs about any of it.