Fork me on GitHub
#tools-deps
<
2020-02-24
>
onetom05:02:50

I'm trying to switch to Google's Maven Central mirror. I've added a mirror section to my ~/.m2/settings.xml but I'm not sure if it's used by clj. How can I check what actual URL is it downloading from when it says:

Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar from central
I've tried -Sverbose and -Sdescribe but neither of those options provide an answer

Alex Miller (Clojure team)13:02:07

I don’t think there is any way to see that, but there is code with the intent to use mirrors in settings

Alex Miller (Clojure team)13:02:42

Someone else from Asia walked down this same path a few months ago and it was working

EmmanuelOga06:02:33

perhaps you need to also add it to the deps.edn map?

onetom06:02:35

I'm trying to move it out from my deps.edns becaues mirror settings are not project specific

EmmanuelOga06:02:14

what if you put it in the global deps.edn?

EmmanuelOga06:02:46

$HOME/.clojure/deps.edn

onetom06:02:14

then pure maven tools wouldn't benefit from this setting

onetom06:02:52

plus im not sure what's the semantic difference between overriding the repo called "central" vs specifying mirrors instead

onetom06:02:36

it's also a bit unclear how the :mvn/repos key is merged into the hardwired default values, because i haven't found a way to print the effective value of it using clj

onetom06:02:27

i had the impression that the :mvn/repos key in my project's deps.edn overrides the value coming from ~/.clojure/deps.edn but i haven't found this being documented

onetom06:02:46

the only reason i suspect that the <mirror> setting is taken into account is because the installation of my deps from scratch takes 34s instead of 51s

EmmanuelOga06:02:03

are you working with a setup that for some reason needs toperform installs from scratch often?

onetom07:02:11

im in hong kong and im on crappy mobile connections often, so using mirrors make a big difference even if im not doing from-scratch installs. on the other hand im also experimenting with building self-contained developer workstation setups and then i want to test from-scratch builds sometimes just to make sure everything still works. a datomic upgrade also pulls in quite some dependencies and our projects might be on slightly different versions of common libraries and all this adds up... just to demonstrate the absolute scale of problems: • im taking a 30minute ferry ride twice a day to work and i could spend about 20minutes of that working • i might need to run a nix package or maven update every week probably which would amount to downloading ~200MB. that can take 10minutes; half of my travel time is wasted. • if i use a mirror and it just takes 5 minutes to upgrade, only 1/4th of my travel time is wasted. but generally many things are significantly slower from asia, so even if i run something on a singapore/tokyo/hong kong aws or digital ocean server, the download time differences add up to minutes compared to US servers.

onetom07:02:43

if u consider that multiple ppl in the company suffer that time waste from time to time, then it's a non-negligible factor

onetom07:02:17

and then i haven't even talked about ppl in china-proper, behind the GFW, who often have to go thru very slow vpns. for them an asian mirror might be several times faster or not even being blocked, so they dont need a vpn to access...

EmmanuelOga07:02:15

oh wow ok setting up that mirro rsounds like a worthy goal

onetom07:02:18

because of these circumstances, im very excited to see parallel downloads in tools.deps! now it performs just as fast as npm2/yarn/pnpm for example 🙂

onetom07:02:47

one less critique i have to listen to from former JS ppl whom im trying to teach clojure to...

EmmanuelOga07:02:44

would it make sense to setup some sort of system wide caching proxy? quick google found this thing: http://proxychains.sourceforge.net/

EmmanuelOga07:02:14

I imagine there could be a way to make it so every time you open an http connection in any program on your system it would go through your proxy

EmmanuelOga07:02:36

I'm not very good at this kind of thing though so maybe it is just a stupid idea hah

onetom07:02:34

proxychains doesnt do caching, but yeah, it would make sense to setup some proxies, but every ecosystem needs different kind of proxies and then some of the deps we need are private because they are our own, in-house code, or it's datomic. setting up proxy for such an environment is not trivial at all.

onetom07:02:34

ive experimented with various maven mirroring solution, like jfrog and artifactory, etc, but that doesn't solve brew, macos, git, nix, npm caching...

onetom07:02:56

for proxychains, i've actually made a nix package:

{ stdenv, fetchFromGitHub } :
stdenv.mkDerivation rec {
  pname = "proxychains";
  version = "4.14";

  src = fetchFromGitHub {
    owner = "rofl0r";
    repo = "proxychains-ng";
    rev = "v${version}";
    sha256 = "03wk2xpxwc7kwlq6z9jf9df1vlh6p0dn0kf826fh1k7nfaa8c4py";
  };

  postPatch = ''
    # Temporary work-around; most likely fixed by next upstream release
    #sed -i Makefile -e '/-lpthread/a LDFLAGS+=-ldl'
  '';

  meta = {
    description = "Proxifier for SOCKS proxies";
    homepage = ;
    license = stdenv.lib.licenses.gpl2Plus;
    platforms = stdenv.lib.platforms.all;
  };
}
and using this config file for it:
[ProxyList]
socks5	127.0.0.1 1080
u can just ssh -D 1080 some.aws.basition.host to establish a tunnel, then
export MYSQL_HOST=
proxychains4 -f proxychains.conf mysql --host=$MYSQL_HOST --user=root --password -e 'show databases'
would allow accessing an RDS instance using its internal aws hostname, although the mysql cli tool doesn't support socks proxies. just showing, to clarify what proxychains is for

onetom07:02:03

also, this kind of proxying over SSH is also said to be disrupted across the GFW. i find it a bit hard to believe, but that's what i've heard...

onetom07:02:39

while all this might sound off-topic on this channel, i think it's actually not, because these kind of scenarios are the reasons, why supporting various directives in maven settings.xml or providing parallel downloads is important. it's just ppl suffering from these issues don't really have a voice to raise them, because they don't speak good english and they are not aware where should they speak up...

👍 4
EmmanuelOga07:02:37

would you say clojure is popular in hong kong?

onetom07:02:52

well, im personally making sure it is ;D

😁 12
EmmanuelOga07:02:29

nice. Good luck with that 🙂

onetom07:02:45

i've introduced clojure into almost all companies i've worked at in the past ~5years https://www.linkedin.com/in/onetom/ im not going to meetups often lately, but when i do, i also sow the seeds of clojure amongst the audience 😉

EmmanuelOga07:02:46

do you speak chinese and/or cantonese?

onetom07:02:54

nope, only a bit of thai, but not at a technical level; im hungarian. i've started to shake up the hungarian clojure community lately too on #clojure-hungary for example