Fork me on GitHub
#tools-deps
<
2022-03-15
>
Adam Helins15:03:56

The company I currently work for has deployed great efforts trying to make clojure work in the context of Nix. Hours and days spent on it by talented ops folks and it turned out to be a real rabbit hole met with failure. I believe @ericdallo has encountered unresolved difficulties as well. The core problem is that we have a really hard time reaching a situation where clojure would work without any access to the internet, for the purpose of doing a "pure" build. That is, even when "everything" seems to be available locally (libs in local maven repo, gitlibs, etc), it still needs the network for some unknown reasons. Quoting @taeer.bar-yam :

We have actually gotten it to work without internet access

This is part of what's so confusing

We can get it to work after running `unshare -n`

So there is clearly some collection of files that when present will convince it not to go to the internet

We just can't figure out what set of files that is

👍 1
ericdallo17:03:47

Maybe @UFDRD93RR may be interested on this as he knows a lot of Nix

kokada17:03:49

AFAIR when I tried to compile something using tools.deps from scratch in Nix

kokada17:03:22

At least looking at all available documentation, I didn't found a way to do a completely offline build of it

kokada17:03:16

In leiningen I remember that I got something to work (I don't remember if it did actually compile and run successfully, but I think it did)

👍 1
kokada17:03:33

But tools.deps seems to be lacking some way to do a completely offline build

kokada17:03:42

https://github.com/NixOS/nixpkgs/commit/2c3ad4ef9c5715fd5b74b4a664b54f5ab8ac7a2c Yeah, I am right. I got clojure-lsp to compile successfully from source with leiningen

kokada17:03:27

So yeah, I may be wrong but I think the problem is with tools.deps

kokada17:03:55

It would be nice to have a flag on it to do a completely offline build (and also another flag to use a custom Maven location instead the default one)

ericdallo17:03:36

Agree with that ☝️

Wanja Hentze18:03:58

Here's some things we did try: • As mentioned, pre-filling .m2/repository and .gitlibs - both via fixed-output derivations and also just taking them from my home directory (after building normally, to warm the cache) as-is. (doesn't work, it still wants to fetch package manifests) • Taking .cpcache as-is in addition to the above (same problem) • Patching everything in .cpcache that refers to local directories, pointing it at our prefetched deps instead (still no luck) • Passing a really long generated -Sdeps parameter to add an alias that sets every deps to a :local/root pointing at prefetched deps, then putting that one last in the list of aliases passed to clojure (actually seems to work for clojure-source deps, but not in the general case for jar deps; some of them have pom files demanding they be placed in a specific location relative to their parents, it seems) • Creating a settings.xml telling maven to run offline (doesn't help, it just fails slightly earlier when it realises it would have to talk to the internet)

Wanja Hentze18:03:23

• probably more things that only @taeer.bar-yam knows, I wasn't there for all of it

Wanja Hentze18:03:18

@UFDRD93RR fwiw, giving a different maven location is relatively easy, you can pass -Sdeps '{:mvn/local-repo ".deps/m2"}' The issue is that whenever it doesn't find what it needs in there, it will still fetch from remote repos, and we have been unable to debug why exactly it makes this decision

Taeer Bar-Yam19:03:07

I also pre-populated .config/clojure/deps.edn and .config/clojure/tools/tools.edn (as well as the .clojure version of those files. Importantly, I can build, and then run unshare -n (cutting off internet access) and then build again, and it's fine. When I build that second time with strace, I see that the only files it access are files that the hermetic nix build should have accessible to it (once we add in all the files we've added in).

Taeer Bar-Yam19:03:51

and when I run strace inside the hermetic nix build, it's not trying to access any files that it can normally but can't in the nix build

kokada20:03:59

@U033F53LU0M yeah, I know. This is exactly what I tried, and this resulted in the same problem you're describing.

kokada20:03:11

Ah, you're talking about the custom Maven flag. Yeah, I am pretty sure that I saw this too and I was just misremembering that tools.deps didn't had way to do

kokada20:03:59

BTW, I remember doing pretty much all the things that @taeer.bar-yam said too. Except for the strace part, I pretty much simply gave up eventually

😩 1
Adam Helins10:03:18

@U064X3EF3 Regarding a repro case, we have a private codebase but we'll try to figure out a way to share the relevant bits. In light of all the above, what has been tried, do you maybe have some suggestions on what would be most useful to you? Reducing the scope of the repro in a meaningful way?

Alex Miller (Clojure team)12:03:46

In regards to a “pure” build, what are you actually trying to do with clojure? Run a repl, start a program with -M or -X or -T? What features are you using in the build? Maven deps? S3 maven deps? Git deps? Local deps? The latter two with deps.edn? pom.xml?

Alex Miller (Clojure team)12:03:06

Ideally I just want a smallish case that illustrates this issue in your env - the deps.edn and the commands you are running

👍 1
Adam Helins13:03:45

A very good starter would be to build an uberjar using tools.build . All attempts were trying to prefetch and prepare everything needed in advance (`tools.build` + deps for the uberjar itself) so that we could clojure -T completely offline (which is precisely what fails). Deps consist of Maven deps mostly, a couple Git ones, and 1 :local/root with a deps.edn.

Alex Miller (Clojure team)13:03:01

can you repro this with like a simple example from the tools.build guide? filing an ask.clojure question would help me keep track of this. I'll be out most of the rest of the week.

Adam Helins13:03:56

Yep, we'll get something out and I'll follow up on ask.clojure Thanks 🙂

Alex Miller (Clojure team)14:03:17

and in case it's not clear, I do want to help you succeed at this - there should be a way to avoid hitting the net if we have what we need. I assume the issue is in canonicalization, either in resolving version ranges for maven or checking git sha/tags or something like that in git libs

🙏 1
ericdallo14:03:00

Really excited to see that working 🤞

kokada17:03:18

BTW, since we are already talking about doing reproducible Clojure builds in Nix, it is important to remember that we don't build Clojure/Leiningen itself from source, instead opting for packaging the release jars

kokada17:03:02

It would be very interesting that we could build it from source instead, so we could trust the result binary being compiled from source instead of trusting the release binaries from upstream

☝️ 1
Alex Miller (Clojure team)17:03:41

well, you can, although I don't know why you'd trust that more than a signed binary

Alex Miller (Clojure team)17:03:58

debian team does this already

kokada21:03:44

For one we don't check the binary signature (only the checksum). Probably we should, but not sure how/if this is possible on nixpkgs

kokada21:03:21

But I still think we should build from source if possible. This is the only way to really guarantee that the whole system is reproducible

Adam Helins13:03:49

Let's say that would be a second step, if we ought to prioritize 🙂 But it does feel like a direction since there is a trend evolving towards fully reproducible builds.

Taeer Bar-Yam16:03:55

Okay, I made [the reproduction](https://github.com/radvendii/clojure-tools.build-test) In doing so, I discovered that the problem doesn't happen with old versions of clojure. In particular the version upgrade from 1.10.3.855 -> 1.10.3.933

Taeer Bar-Yam16:03:59

I looked at [the changelog](https://clojure.org/releases/tools), and I'm wondering if the problem is tools, and in particular them getting fetched in a different way. Is there a way to disable that feature, to test that out?

Taeer Bar-Yam16:03:24

Alternatively, if we know where the tools and tool information gets cached, we could try providing those files to the nix build

Alex Miller (Clojure team)15:03:25

can you file a reproducible case at https://ask.clojure.org ?