Fork me on GitHub
#tools-deps
<
2024-07-06
>
Mark Wardle13:07:55

Hi all, I think there is a potential issue with the use of a proxy and downloading dependencies behind a proxy in a corporate network..... but providing a reproducible environment to demonstrate is difficult, so all I can do is explain what happened.....

Mark Wardle13:07:13

I was trying to run a Clojure project from source using -X on a server that can only access the internet via a HTTP proxy. This is a linux server, with environmental variables and a maven .m2/settings.xml setting the proxy to use correctly. Just in case, I also added the -J-Dhttps.proxyHost and proxyPort settings to my command line. ie : I'm choosing the same proxy settings in 3 different ways: maven settings, -J command line and env variables.

Mark Wardle13:07:20

When resolving dependencies on clojars, or using maven install using a pom.xml created on a machine not behind a proxy, dependency resolution worked. However, the Clojure cli tool could not seem to pull down dependencies itself, getting a timeout.

Mark Wardle13:07:13

The only fix was for me to build a pom.xml on another machine on another network:

Mark Wardle13:07:18

$ clj -X deps mvn-pom

Mark Wardle13:07:03

And then copy that file to the protected machine, and run:

$ mvn dependency:resolve
or
$ mvn install

Mark Wardle13:07:20

Further invocations of clj then would use the local .m2 cache and work fine.

Mark Wardle13:07:24

It therefore appears that in some way, tools.deps isn't using the proxy settings set either from the maven configuration or from the java command line (or from the environmental variables, but not sure any JVM project does that anyway).

Mark Wardle13:07:52

I was hoping to be able to provide a minimal reproducible example, but haven't been able to do so outside of the live environment.

Mark Wardle13:07:22

TLDR: if you are stuck behind a proxy and cannot download dependencies, try generating a pom.xml and using maven instead. TLDR: there may be a bug in tools.deps that means that at least one of the network calls does not use the proxy settings, such that the call fails with a timeout. I can't prove this however.

Alex Miller (Clojure team)13:07:31

It should be using the maven settings.xml proxy settings

Alex Miller (Clojure team)13:07:08

Can you tell if it depends which maven repo (central vs clojars)?

Alex Miller (Clojure team)14:07:06

The -J won’t do anything - that affects your program jvm, not the jvm that downloads deps. You can set those settings in CLJ_JVM_OPTS env var if you want to try that

👍 1
Mark Wardle14:07:13

Yes I think you are right in that it worked for some dependencies, but not all, but I'd have to be back in the office to double check which way around it was. I also specify another third-party repo in my deps.edn so it might be one or more of the three. All downloaded when using maven directly. I can try again and see if I can get a more verbose debug output when I'm back in office. Didn't know about the CLj_JVM_OPTS option but will try that as well.

jumar15:07:28

The proxy doesn’t enforce any authentication?

Mark Wardle15:07:43

Yes it does. But I get a timeout not an auth failure and the key thing is that maven works so I have my .m2 config correct. I do recall it logging downloads from at least one repository but I’ll do more sleuthing as I can’t recall off hand which appeared to ‘block’.

Alex Miller (Clojure team)15:07:38

maybe it's just taking too long via the proxy?

Mark Wardle16:07:30

I will do a bit more investigative work and check it isn't stalling with a specific repo but that still wouldn't explain the fact that maven works.

dominicm17:07:38

Unless the timeout is configured differently in maven? :thinking_face:

Mark Wardle17:07:19

Pretty sure the maven process looked like it was downloading the whole internet (as it usually does) very quickly but I will double check.