Fork me on GitHub
#polylith
<
2022-02-23
>
pavlosmelissinos08:02:35

We're about to switch to a monorepo but there's some reluctance in the team to try out polylith and I'm wondering if I should try to champion it or settle for the "dumb"/simple option (a single "projects" root folder) Some arguments I've heard are: • Discoverability will take a hit and search will be harder; polylith forces you to break down logic into multiple files when before you could have most of the logic in a single namespace - the merits of splitting code are not immediately apparent if reusability isn't a concern (e.g. code is only used once) • Sounds a lot like Java's OSGi (no experience with it but it does sound a bit like polylith sans the ability to develop with all bricks in the classpath) I understand the reluctance and I'm still not ready to 100% bet on polylith because it requires significant time investment in the longer term, so I'd like your help. What made the scale tip towards polylith in your case? When did you know that it was the right thing for you? What drawbacks does it really have? What is its biggest strength? I appreciate the polylith team's perspective and I'd be glad to hear it but for this I'd like some input mostly from the users of the tool. 🙂 Thanks a lot

imre08:02:05

> polylith forces you to break down logic into multiple files I don't think this is always correct. You can have all code in the interface namespace if you want to.

Teemu Kaukoranta08:02:20

or just have a big base? (Polylith newbie here 😄 )

pavlosmelissinos09:02:23

The problem isn't so much which namespace within a brick to put your logic in, it's more the fact that there are bricks in the first place. Say you have a monolith with 10 big namespaces (1K locs each). Your options are either to break it down into multiple bricks (components/bases) or have a couple large bases. The first option kind of hurts discoverability, the second doesn't offer a lot of value

Teemu Kaukoranta09:02:07

Ahh, it being a monolith is certainly a key consideration here

imre09:02:02

I suppose polylith's value add is when you are trying to break a monolith down, presumably for component reusability

imre09:02:55

But I also don't see a problem having some of your projects in the monorepo being more monolithic than others

👍 1
Teemu Kaukoranta09:02:27

Other advantages is that Polylith will solve how to only run tests / build only parts that have changed, how to have a single repl, how to share code when you eventually want to (I'm sure there's some parts you'd like to share)

👍 1
Teemu Kaukoranta09:02:33

@UEQPKG7HQ some more context could be helpful if you care to share: how big is your team (and how experienced), how many repos you have, how many lines of code. What are the problems you're trying to solve with a monorepo?

pavlosmelissinos10:02:34

> how big is your team (and how experienced) Pretty small, ~4 people mid and senior > how many repos you have we're considering merging at least 3 repos together, A is about ~15K locs, B is 1K and C is really small (without tests, the numbers are doubled if I include tests) > What are the problems you're trying to solve with a monorepo 1. Right now we have to sync the releases: A is a dependency of B and C (in their deps.edn) so whenever there's a new release for A we have to bump its version on repos B and C manually and trigger the individual releases 2. The repos are too coupled to be separate: project C is an API that exposes some of the functionality of A, so adding a new endpoint to C sometimes requires a separate PR to project A If you ask me, Polylith seems like a good option but I have to convince the rest of the team (or have my mind changed) - I definitely don't want to be proven wrong after we've switched 😄

Teemu Kaukoranta12:02:05

So is A used as more of a library, or do you actually have A, B and C running in production? As separate artifacts?

pavlosmelissinos12:02:28

They are all separate apps but A is also used as a library by the other two. It's a bit more complicated in practice, as always, but that's the gist.

Teemu Kaukoranta13:02:39

So, I'm probably not the best person to answer your question, since we're still in the process of adopting Polylith. But then again, I've recently been thinking about the same things that you are now thinking about, so maybe these are relevant. First, you and your team should take a look at Sean's blog if you haven't already. https://corfield.org/blog/2021/02/23/deps-edn-monorepo/ One big thing, to me, in favor of Polylith, was that I realised that if we don't adopt Polylith, we will probably end up recreating parts of it. I'm referring to the stuff that I mentioned already, like running tests for only the parts that changed. Certainly you can setup this stuff yourself too, but is it worth it? A lot of people are anxious about splitting code into different files, but it is something you get used to. The polylith model sounds like a good fit for you btw, you could have bases A, B, C, and you could split off the reused parts of A into a component.

👍 1
seancorfield17:02:14

@UEQPKG7HQ Happy to chat about this topic in detail since we've had a monorepo for years and have been migrating to Polylith for quite a while. We have about 130K lines of code in 18 projects and at this point about 40 components, and a dozen bases. Some of those projects don't yet have corresponding bases -- they are built from legacy subprojects (we still have about 30 legacy subprojects). For us, even just separating the artifact building into projects and the main APIs/CLIs into bases has shown value, and the focus on small, deliberately named components has really helped us think about our code organization. Feel free to DM me with Qs if you want, where I can go into more detail about our code than I might feel comfortable doing in a public channel. Also happy to answer general Qs in public, of course.

👍 2
Pieter Koornhof18:02:02

Quite a few of our developers have also echo’d the fact that poly forces them to stop and think about the interfaces they make available to the rest of the code base. If you think about it there is nothing that really forces that other than files being named interface. You can apply the same reasoning to any codebase, but for some reason we never did that consistently and now we do. Also you can create a bunch of custom tooling around poly and the workspace.edn if you want. I have a network diagram that quickly shows component relationships and potential problems etc. Discoverability is no more of an issue than it was pre poly. It’s also important to note that you can just start off with 3 bases and then slowly move into components as it seems logical. We are slowly expanding and refactoring into components only when those bits of code needs work. Adopting poly does not mean you have to refactor everything into component before you get the benefit, instead you can just have what you always had, with the benefit of now moving forward with poly. We had to overcome quite a few unique problems, but it still feels like poly is the correct platform for us to move our code base forward. 4 bases (3 of which have cljs apps), 29 components ~75k loc

1
👍 1
Pieter Koornhof18:02:55

as a side note the code is now the most repl driven development friendly it has ever been

seancorfield18:02:05

"poly forces them to stop and think about the interfaces they make available to the rest of the code base" -- yes, this, so much! And a focus on naming.

☝️ 1
pavlosmelissinos18:02:37

Thank you all for sharing your experiences, I really appreciate it. 🙂 > You can apply the same reasoning to any codebase, but for some reason we never did that consistently and now we do. I think this is key. In theory you don't need polylith but in practice teams are rarely disciplined enough on their own. > Feel free to DM me with Qs if you want That's actually awesome @U04V70XH6, thanks! 😄 Everything is still quite vague and I don't want to waste your time. However, I'll speak with the team and I might come back to you with some more focused questions in the next couple of weeks if that's ok with you!

Hukka08:03:55

I somehow feel that polylith doesn't make sense, until you have already decided to do what polylith does. At that point poly gives you tooling. In other words, you can do monorepos, or splitting monoliths into reusable libraries/components without poly, and for good reasons. At that point polylith gives you vocabulary to talk about things, tools and consistency. It could even help with splitting a monolith without a monorepo, if you start goofing around with submodules and such. But there's nothing really special about polylith in the sense that much of the arguments for it are arguments for doing it manually too. I guess it would be a rare case where a team thinks that a monorepo with split components makes sense, but they are not going to do it for the lack of tools.

Hukka08:03:42

In your case it really does sound like some parts of A should be split of. Doesn't sound very sensible that A is in one big namespace that can work standalone, but also some of it are used in another "standalone" project. But again, if the team is not willing to split anything, then poly tooling is not going to give a lot of benefit compared to just having a monorepo with three folders. Maybe just start from the monorepo, since you are feeling the pain of syncing repositories? And if after that splitting does feel sensible, go with polylith conventions and tooling?

Hukka08:03:50

For what's it worth, I don't think polylith > requires significant time investment in the longer term since you can do it piecemeal, starting from huge namespaces, and it would take at least as much investment to do the splitting without it

Hukka08:03:24

In fact, you don't even need to have the whole monorepo under polylith "control". We have one pretty separate backend project that doesn't share code, and the whole frontend in the same repo without any problems. With the frontend the benefits of monorepo are definitely there since it's trivial to sync API changes.

seancorfield16:03:48

> I somehow feel that polylith doesn't make sense, until you have already decided to do what polylith does. At that point poly gives you tooling. We'd gone through several iterations of our monorepo before adopting Polylith so I kind of agree with you here: what we needed at that point was some formalization of our practices, code organization, and naming -- and Polylith brings all of that with tool support. We're still very much in a hybrid model with 28 non-Polylith subprojects yet to be migrated. We have 18 projects, 11 of which are built from bases (the other 7 from "legacy" subprojects). We have nearly 40 components at this point and still a lot of namespaces to refactor from "legacy" subprojects).

Nikolas Pafitis12:02:31

Hello guys, I don't know if this has been discussed before, but one issue I often find myself encountering is having to restart the REPL everytime i create a new brick. In the case of a traditional project structure with src/ as the root of the source path, any new namespace you create will be under the existing path and you'd be able to load the namespace into your running REPL. but with polylith and the dev workspace, if you create a new brick you'll have to update your paths and restart the REPL. How do you guys manage this?

furkan3ayraktar13:02:47

I load the new namespaces into my REPL and it works well without needing to restart the REPL

Nikolas Pafitis13:02:10

Even if the new namespace is not under path?

pavlosmelissinos13:02:12

You don't have to restart the REPL to load new namespaces to it, doesn't matter if the path to the actual clj file is in the classpath or not.

pavlosmelissinos13:02:14

also if you want to add a new dependency to a running REPL you can use the add-lib3 branch of tools.deps.alpha

👍 1
mindbender10:02:08

@UEQPKG7HQ Which function exactly in the add-lib3 branch of tools.dep.alpha can be used to add the dependency to a running REPL?

hiredman18:02:47

speaking of interface files, I can't help but feel sort of like we just got off the core.clj train, and now we have interface.clj everywhere, isn't there a better way than having a ton of identically named files?

1
seancorfield19:02:28

I advocated for the ability to have <top-ns>/<component>.clj as the "interface" (and then the implementation in <top-ns>/<component>/<whatever>.clj but it would add a lot of complexity to the poly tool to support both approaches (which it would have to do for backward compatibility or even to permit a migration within an existing Polylith codebase).

imre08:02:12

I would also prefer the approach Sean described here but it isn't a dealbreaker fortunately. Would still be nice to have that option when creating a brand new workspace.

seancorfield19:02:28

I advocated for the ability to have <top-ns>/<component>.clj as the "interface" (and then the implementation in <top-ns>/<component>/<whatever>.clj but it would add a lot of complexity to the poly tool to support both approaches (which it would have to do for backward compatibility or even to permit a migration within an existing Polylith codebase).

seancorfield23:02:55

What does this warning mean?

Warning 204: The same library dependency exists in both the development project and the default profile: org.clojure/tools.logging
Lots of my bricks depend on that library, and it just so happens that the component implementation specified in :+default now has that dependency too. Why would this be a problem? Or is it a bug in the poly check function?

tengstrand05:02:36

Can you export your project with e.g. poly ws out:sean.edn and mail it to me, and I will have a look @U04V70XH6.

tengstrand06:02:59

It looks like you have org.clojure/tools.logging explicitly defined in ./deps.edn . If you remove it from there, it will be included in development anyway.

tengstrand06:02:02

An alternative is to remove it from the defult profile, instead of adding it to both the default and httpkit profiles.

seancorfield06:02:47

It is not in ./deps.edn

tengstrand06:02:22

Okay. Then this is a bug. Can you create an issue?

tengstrand06:02:01

Can you email your ./deps.edn file also?

seancorfield06:02:05

I assume it's because :dev brings in bricks that depend on tools.logging and the :+default alias also brings in a brick that depends on it.

tengstrand06:02:40

That’s my guess also. This worked fine before we added support for :local/root I guess.

seancorfield06:02:56

Sent. Sorry for the non-attachment version 🙂

seancorfield06:02:06

We're at:

projects: 18   interfaces: 38
  bases:    11   components: 39
...
(lines) 33,195 3,743
now!

tengstrand06:02:48

Yes, and the output from libs was quite impressive!

seancorfield06:02:00

Hahaha... yeah, it's a big project.

seancorfield06:02:20

And we're only a third of the way to migrating to Polylith 🙂

👍 1