Clojurians Log v2

Hmm, that would indicate a problem with a Fastly node I think. Either it recovered or you got a different node on restart.

Thanks for letting me know!

yeah, I don't think we can glean anything from it, but here's the line that downloaded it successfully on the restart https://github.com/typedclojure/typedclojure/actions/runs/15450851600/job/43493290964#step:6:81

the correspondence is not exact (jar has maven files and manifest), and sometimes maven filtering will modify resource files, pretty common in java jars less so in Clojure. but I think you could probably parameterize what should be content identical

github now has support for https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds (SLSA is the standard people are using for this). I think we could be building that stuff directly into our packaging tools and then look at verification paths better than the unworkable gpg stuff we have now

the big problem with the gpg signatures currently used in maven repos is that a producer can sign an artifact, and a consumer can verify that it was signed, but consumers have no way to know whether the key used for signing is actually associated with the artifact builder - that part requires a consumer to manually verify that the public key owner is associated with the project, and how many projects are clear about what that is? (very few - but see Clojure's https://clojure.org/releases/download_key). I don't know a lot about the Sigstore stuff github is using, but if it's integrated well with your github account, I assume it's better sorted than the gpg key mess

I would love to make it easy (automatic even!) to create and verify artifacts with signed SBOMs with our standard toolset in the Clojure community to reduce the risk of supply chain attacks. Phil and I have had a few chats about this.

And the tools landscape is not limited to tools.build and maybe not even github.

> the correspondence is not exact (jar has maven files and manifest), and sometimes maven filtering will modify resource files, pretty common in java jars less so in Clojure. agreed. good point about the maven-related files, still thinking that through. my first rung of reproducibility that I was hoping to achieve was determining the provenance of every clojure file on the classpath. Or at least a list of clojure files whose provenance could not be inferred, or are shadowed by AOT sources. I think my idea would be most effective on projects that also work as git deps without a prep step, which would exclude projects that modify resource files, so yes, agreed this won't help. The reasons I find this simplistic idea interesting: • I think many clojure libs fall into this category, so there's a good return on investment • no signing or build reproductions, but still seems like a strong result (effectively content addressed jars) • it's cheap to reproduce independently. it could be used to generate expected checksums for such jars that you then check in to your repo that a build tool could use. • if you're lucky, you could apply a variant of this idea retroactively to scan old jars. e.g, maybe source deps.edn from the repo, or even just match any file

Here's an interesting and achievable combination of the above ideas: for each jar in clojars that claims a git sha, run a build whose https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds is a report of the provenance of each file in the jar vs the sha. The clojars page for the jar then presents relevant results ("95% verified provenance for .clj files") and links to instructions to verify the artifact.

A dependency resolver could then verify provenance without needing to clone anything. It asks clojars for the report for each jar and automatically verifies the attestation (presumably via a trusted reusable workflow). Then aggregates the reports for the dep tree and presents it to the user (e.g., fail the build on unknown .clj files, ask the user to exclude files / find provenance manually).

If I understand artifact attestation correctly, a user should be able upload their own attestation to clojars too, as long as it uses the same trusted workflow.

The verification procedure seems the same regardless of who triggers the workflow. AFAICT:

gh attestation verify -o clojars --signer-workflow clojars/attestation-workflows/.github/workflows/verify-provenance.yml provenance-report.edn

Hmm actually the -o clojars is too specific for that scenario. I wonder if you can just specify the signer workflow. I guess clojars could also send any extra metadata to help verify the attestation.

To decouple this from clojars altogether, the attested provenance report could also be embedded in the jar itself, but it would require some extra juggling of attesting the artifact before pushing the jar. Clojars would still be involved by providing the trusted workflow but the jar can now be deployed elsewhere.

you're basically working your way back to SBOMs and SLSA, why not just implement it

yeah, definitely converging but I think there's a meaningful difference. the picture isn't clear yet but: 1. In this line of thinking, individual maintainers don't need to be involved beyond adding scm.tag to the jar. 2. Verifying attestation is automatable. there's one trusted workflow you need to verify manually. 3. Supporting existing jars can be automated. 4. With this in place, the community can concentrate on the projects that really need custom workflows with non-trivial builds. Verifying attestation might be harder for these, so the less the better. 5. We could plausibly infer missing metadata to verify even poorly built jars. e.g., If there's just scm.url, we could clone the repo, and search for a commit with the same files.

I guess I'm saying we should definitely implement SBOM verification, but this idea might be a supplementary check to ease the maintenance burden on the community for simple projects.

my open questions of just using SLSA: 1. what is the maintenance burden of verifying attestations? do you need to understand the build + workflow of each dependency? 2. do you get provenance in the end of individual clojure files? I think only if you completely understand the build, you can infer it. same for the "trusted clojars workflow" but you at least only need to understand one workflow and can ignore any e.g., tools.build steps. 3. how do we get maintainers on board to attest their artifacts

all necessary problems to address but maybe we can skip all those questions for a subset of jars with this idea.

dunno, this is below the threshold of what I've looked at. my hope would be that a) build systems would make and sign the SBOM etc with low effort on maintainers part so it becomes easy and standard to do, b) repos (Maven, git, whatever) could store the necessary files either in or alongside jars/releases, and c) the tools we use have can automatically verify that the artifacts in our supply chain are valid and as intended with support for auditing. The last step is interesting b/c I assume verification takes time and properly mitigating that or choosing when things get (re)verified is a continuum with tradeoffs.

we're pretty much on the same page in terms of goals.

maybe I'm just zooming in on a subset of projects that don't really have a "build step" (or it's fairly transparent), and seeing if there's a head-start to be made while all this infrastructure is set up.

I think a) is the weakest link, so it's attractive to think about opportunities to remove it.

i.e., having maintainers involved at all

other reasons a) is hard: 1. it's probably a difficult problem to make the process low-effort 2. unlikely to be retroactively applied to past versions

the essential insight I keep coming back to is: IIUC we don't need all this ceremony to verify the supply chain if our entire dep tree is git deps. why is that? can we reuse those same reasons to automate verifying some jars?

in terms of long term strategy, it would be nice to encourage the community to distribute libraries using source-only jars so most dep trees end up looking just like a tree of git deps. maybe if there are enough advantages (e.g., zero involvement in SBOM's for maintainers), people will ask for specific enhancements to the clojure compiler so there's less need for build steps => less bytecode flying around to verify.

if that works out, manual SBOM's would then only be needed for big apps, and then their own supply chain attestation verifications would be much easier (presumably they're using open source libs from source-only jars)

it would be a virtuous cycle (in terms of supply chain auditability) of app builders demanding libs have easily verifiable attestations further discouraging (unnecessary) bytecode from being packaged with libs.

source-only is already the recommendation for libraries

and I think that's the 98% case in practice right now

clojure itself is kind of the exception there

there you go. can we automate the provenance of that 98% without even notifying their maintainers? 🙂

yeah those big projects like clj/cljs are definitely needing the full treatment.

and we could also search github for prep steps for ideas on whether compiler enhancements could bump that higher.

but that's extra credit.

> The last step is interesting b/c I assume verification takes time and properly mitigating that or choosing when things get (re)verified is a continuum with tradeoffs. agreed. this is one way that establishing source provenance to sha's (if it is deemed sufficient) has an advantage, because it could be crosschecked fairly safely and quickly (compared to a full build). or even be completely replicated at key times like deployments, and remove needing to trust the attester altogether.