Clojurians Log v2

Partially related to this thread: https://clojurians.slack.com/archives/C6QH853H8/p1738449167894009 It's mostly to allow dependency version bumping tools like Renovate to work with major version branch strategies and avoid the extra process overhead from using Git tags + releases. Just :git/sha doesn't work well because auto-updaters can't tie break among multiple descendants. Descendant checks without a branch also means feature branches off the tip of the default branch will accidentally be considered by the auto-update tool.

I don't see anything on https://ask.clojure.org/ so you could write up a question about it there.

It’s intentionally not supported because branches are not bound to a single commit

It would have to work like tags where you can't specify only a branch, rather a branch + SHA must be used.

Any Git ref (tag or branch) has to be paired with a SHA to lock it down (tags can be deleted + recreated, branch tips will accrete new commits).

Ask Clojure post for ref: https://ask.clojure.org/index.php/14400/deps-edn-support-for-git-branch-git-sha

The primary here is the sha, the tag is used for semantic verification. While (some) tags can be deleted and moved, that is not normal practice - generally the intent of a tag is to name a single commit, vs branches whose intent is to serve as a mutable identity, and is thus contrary to the purpose of specifying a single commit

If you don’t want to use a tag, don’t - just use the commit sha

Can't a branch name also be used for semantic verification as well? If we only have a commit SHA, any kind of auto-update mechanism (Renovate, Dependabot) has no proper descendant tie breaking data. That would have to be supplied outside of the deps.edn file which becomes unpleasant if you have a lot of Git deps.

I understand the intent of a tag is a single commit, but fundamentally all Git refs are mutable so I don't see why tags get special treatment over major version branches.

From the Ask Clojure answer, it sounds like there's some other special consideration for deps expansion that aren't an issue with tags but are an issue with branches for figuring out the latest descendant.

the pattern of use between tags and branches are different. while (some) tags can be moved, they typically are not, whereas branches are by design not a fixed point, but yes there are other considerations too.

Hmm then on a slight tangent, are there plans to add a deps upgrade command to the Clojure CLI/`tools.deps`? For Git libs, it seems like it'll force usage of tags to let that feature auto-update Git SHAs. For ref-less Git lib entries in deps.edn, there's no sane way of weeding out PR branches from the default/main/major-version branch. A ref (tag or branch) is needed to prune descendant commit subtrees from consideration.

There are already tools like liquidz/antq for that deps upgrades.

liquidz/antq has an open issue regarding branch selection for upgrades. https://github.com/liquidz/antq/issues/131

Right, but that's your best direction for addressing your need. As Alex explained, you're not going to get branch/short-sha in tools.deps itself.

I'm open to ideas around how git "versions" are found and ordered

we have a strategy now, but perhaps other strategies could be available

maybe it's also useful to consider the use cases you're working on. for example, is this a case where you are developing multiple projects in tandem and trying to track them. if so, maybe local deps are a better match, don't know

I think local deps forces a monorepo which has tradeoffs in requiring adoption of other monorepo tools and making CI complex

it forces a common developer directory structure, but that does not necessarily imply a monorepo

doesn't it require the monorepo to deal with transitive local git deps?

or hmm maybe there's a workaround with Git submodules or subtrees

submodules are not a good match if you are using git deps (because they use worktrees which do not play well with submodules). but if you are using local deps, you only require a standard relative directory structure (all projects with common parent dir is sufficient for example)

just to make sure we are meaning the same thing.... by local, I'm talking about :local/root deps, not local git deps

so each project would do something like :local/root "../shared-lib" and then to build in a polyrepo scenario, something needs to know all the right things to check out to form the build directory

yes

need to ponder that for a bit. Some build tools like Nix have a way to effectively do this via an ephemeral symlink farm, but for IDE/LSP support the checkout of a single project should really pull everything

There's a reason we use Polylith and a monorepo at work for our two dozen services 🙂

I think that's fine at that scale, but it gets hard without dedicated monorepo tooling once you scale past that (e.g. Amazon whose Brazil build system makes the polyrepo setup feel like a monorepo, only special tooling is the build system, Git tools are all vanilla).

How big is your codebase?

couple hundred repos

GitLab in particular is pretty bad at handling monorepos because the merge train becomes really long due to the high merge frequency

that and the .gitlab-ci.yml becomes quite the hairball with that many separate projects in it

do you have "release points" in these projects? if so, how are those known

Essentially every commit to the main branch of anything in the dependency graph is a release point. For example, suppose we have some AWS ECS-based (like Kubernetes, so some container platform) web service which we split into 4 repos: 1. Specs repo a. Models the service's API methods and requests/responses. b. Includes a helper build tool/function using spec-tools.openapi.core to generate an OpenAPI model from a Clojure spec. 2. Backend repo a. Depends on the spec repo to auto-generate server stubs (e.g. some macro to generate multimethods and protocols. cognitect/aws-api kind of does a similar code gen thing). 3. Infrastructure repo a. Depends on the backend repo to build an uberjar that it puts into an OCI container. b. Uses the AWS CDK to generate CloudFormation templates and upload the OCI container into its bootstrap ECR repository. 4. Client repo a. Depends on the spec repo's OpenAPI util function to feed into an OpenAPI client generator (e.g. openapi-generator, fern). When a commit is merged to main on the specs repo, it should spin up auto-update jobs to update deps.edn in the consumer repos. For example, adding a new API endpoint in the spec should regenerate the backend + client.

Now suppose this single web service is composed of several subservices/microservices so changes should auto-propagate through them all (external consumers we still use tags for client releases. Only integration/canary test repos should get the latest unreleased client).

so you really are basically tracking branches, and I assume you don't want to touch all those deps to pick up the changes

this is somewhat at odds with assumptions built into deps.edn (which assumes that if deps.edn doesn't change, then deps haven't changed) - you'd really need to force a recompute (which you can do with -Sforce)

during design we did talk through a use case like this and decided at the time that local deps were really a better match for this case, but I totally get why you want this

we have similar pain points at nubank (version control over 100s/1000s of repos, not a monorepo, don't want to manually update)

FWIW Amazon's able to get away with this because their build system presents a local package registry daemon (e.g. looks like a normal HTTPS Maven/npm/PyPI registry on localhost) which: 1. Acts as a proxy to the company CodeArtifact which is a gatekeeper to Maven Central, npmjs, PyPI, etc. 2. Presents the language-native package published to the local daemon by git dependencies. Then they just have people use something like a lightly wrapped maven, gradle, or npm which uses the local package registry daemon as the default registry. If that existed in the open source world, we could just point the resolver config to that in deps.edn and use Maven deps but that's not something Amazon will ever open source.

So here we are doing these auto-update hacks with Renovate.

git deps as they exist are really targeted for publishing libs or tools for other people to use, not at supporting organizations using many private git repos in building their products, which would be benefitted by a hybrid of features from local deps and git deps