@niwinz: Oh wow! You rock! I was going to offer to try and help with this, but it looks like your commits are pretty much what I had in mind…. i.e. tackle the tools.deps transition first; then consider the bigger questions about whether or not or how to manage the parent/sub-projects. I think the main requirement here is for any changes in sub or parent projects to retest everything. Have you thought much about that bit? There are three main approaches that… Option A: Keep it as it is right now… i.e. rely on manual coordination/local-testing to ensure changes that are going to be merged are fully tested before release. Option B: Have multiple repos; but make CI watch and pick up changes to any parent/sub-project and integrate and test everything. Pushing/scripting this sort of thing is pretty complicated with most modern “repo orientated” CI’s. Option C: Mono-repo, but multiple deployed artifacts. This makes the automated integration/testing job of CI much easier. Option C: is I think the ideal approach in terms of cost/benefit. However the challenge is then deciding how to build/manage the mono-repo. There are several options: 1. git submodules — This keeps the multiple repositories; but provides a way for the parent to consolidate everything. It is widely acknowledged to be pretty awkward to use though. I’ve also done this multiple times in the past, and can attest it is a PITA — though sometimes it’s still the best option overall. Trade offs. Advantages are it preserves github issues and sub-repos, and lets you improve the CI; but still requires a lot of fiddling/awkwardness to use… though if you and the contributors know about it can work ok. 2. git subtree’s in parent project for each sub project, but also keep sub repos separate. This just works most of the time; but the relationship between the projects/repos is unclear to contributors… It does have slightly nicer affordances than git submodules but muddies the relationship a bit, as changes can be made to the parent or sub-project and combined (though you probably want to encourage just one direction/flow). 3. Make a git subtree in parent project for each sub project, but archive the sub projects and sub-repos on github. This preserves the commit histories and iirc shas of the sub projects up to the combination, but you’d probably need to manually copy/migrate the issues/PRs over to the parent repo. 4. Something with git filter-branch — I can’t quite recall the differences between this and option 3; but suspect it’s around rewriting historical commits. I should also mention there is some fancy stuff you can do with git replace, which will let you have rewritten histories in for example the parent-project, but still provide ways to access the full original history in the sub-project, should you need to. I’ve never tried this though. My vote would be for Option C, achieved through either 3 or 4. The main consideration here is whether to rewrite commits or not. There may be some advantages to selectively rewriting commits such that github issue numbers e.g. #3 don’t get mixed up across projects; though obviously this is an extra level of hassle/complication in achieving the merger.

Anyway I’m curious about what you’re thinking here? I’m also happy to try/explore some options if you wish; though it may take me a while to find the time (kids).

agree with you with C, but, i'm completly NO (hehehe) with subtree/submodules, this is real pain with this, the mental overhead on managint it makes this completelly NO-GO

maybe a single repository, and just that

ok I agree on the pain front about both of those. That rules out option 1 and 2. I think option 3 may still be ok though… i.e. you use git subtree as a means of achieving the single-repo; but you don’t manage it that way… and just archive the sub-project repos. Though I could be wrong on that front.

i don't know the right solution here, right now I have tieme for concrete short and very delimited tasks, and everything that makes this tasks simple, better hehe

for now, i go to convert each repo to tool.deps, and release a version. the next task can be, join all in one repository and create some automatizations

maybe we can take example on how reitit is doing with all submodules

reitit uses leiningen in a mono repo

hmm then it can't be a good example xD

🙂

i think simple scripts that updates versions on multiple packages and then executes the jar building on each will be more than enough

on the end, the build script is just a clojure, so it can read the version from some file on the parent directory

on the end, I think it can be very simple without needing a specialized tool

We’re agreed on that. The clojure CLI and tools.deps/`tools.build` solves this, e.g. with :local/root etc. All you need to do is have a top level script that builds the right classpath and runs all the nested sub project tests. Then script deployments from the top too.

i.e. take the builds out of the sub projects and centralise them all at the parent… and just leave the src/`tests` in sub project dirs.

Are there any sub project branches or PRs that you might want merged after the mono-repo job is done?

I suspect retargeting/rebasing them against a potentially different history probably isn’t going to be that hard for most of them — but haven’t looked.

hey, really interested in getting a monorepo-setup for reitit (and malli) with deps, instead of lein.

In terms of monorepo solutions, I know polilyth is a heavy one, and there's another I can't find at the moment which is lightweight

Maestro: https://github.com/helins/maestro.clj

sincerelly, i'm not very convinced on "specialized tool dependency", is the same discussion of small lib vs framework. As less dependencies, better. I think https://github.com/polyfy/polylith is a good source of inspiration because they do everything using tools.build or at least it seems like that

if we can manage all using a single tool that is already common for all users, will make the maintenance more aproachable

not only for me, just for any one who want digg into the library

all buddy libraries updated

Yeah imho all of the sub projects in buddy, at first glance at least, have simple deps and simple needs. I think it makes 100% sense for consumers to choose to have minimal deps; especially for a security library — and suspect the main reason it is carved up as it is across multiple repos etc, is because of leiningen and the lack of a good scripting ability, with less classpath control. Which is now solved in the tools.deps world. leiningens plugins to handle mono-repo workflows were always a bit clunky imho. When you’re working in buddy itself though, I doubt you really need much in the way of isolated sub/project aliases etc.

Either way I’d probably advocate starting with the smallest/simplest thing… and only adopting a “specialised tool dependency” if there’s a clear need.

The main use cases in my mind are: 1. The project maintainers having workflows for deployment and maintaining the project that work for them. 2. Folk like me who want to make some trivial change to the library and push it upstream. I don’t need to worry about deployment; but I do want to ensure all the tests are run across all of the buddy project; and also prove to the maintainers that they pass, and essentially have them associated with a single branch / PR.

Currently 2 is a bit of a faff… I can run the tests locally with lein checkouts etc; but I can’t share that, and prove my changes were tested and will work.