-Sdeps (No such file or directory) could indicate a quote escaping issue. I'd make sure that: • The clojure CLI tool is the latest (not from debian) • CIDER (the Emacs package) is the latest stable release (https://github.com/clojure-emacs/cider/releases) If it persists, feel free to create an issue in the cider tracker

Unrelated, but we have a debian-specific step that can avoid a frequent pitfall: https://docs.cider.mx/cider/troubleshooting.html#navigation-to-jdk-sources-doesnt-work

It should be trying to retrieve this file https://repo1.maven.org/maven2/org/clojure/clojure/1.11.1/clojure-1.11.1.pom - can you access it? If so, can you write to ~/.m2/repository? Might also be good to check java -version

(should be at least Java 1.8)

$ ls -al ~/.m2/repository/org/clojure/clojure/1.11.1/ total 8 drwxr-xr-x 2 omer omer 4096 Dec 27 10:10 . drwxr-xr-x 3 omer omer 4096 Dec 27 09:09 .. $ java --version openjdk 17.0.9 2023-10-17 OpenJDK Runtime Environment (build 17.0.9+9-Debian-1deb11u1) OpenJDK 64-Bit Server VM (build 17.0.9+9-Debian-1deb11u1, mixed mode, sharing)

Seems that I do not have <https:clojure-1.11.1.pom|clojure-1.11.1.pom> How to retrieve it?

I manually retrieved the above .pom file and placed it in ~/.m2/repository/org/clojure/clojure/1.11.1/ Now clj fails to read org.clojure:spec.alpha:jar:0.3.218 (I added underscores). Seems that I have a problem getting all *.pom files that clj needs. What to check now?

Downloading individual dependencies will add many hours (week) of work to use Clojure (and most other programming languages My assumption is that some certificates are outdated in the older version of Debian being used. The current stable version is Debian 12 (bookworm). Suggest updating to Debian 12 to see if these network challenges are resolved Bookworm is working correctly for me (and also by others in the community). Or it could be a network, firewall or proxy issue, but seems like an OS issue rather than Clojure or Java install.

Agreed, I think something like that is a good guess, although the carts should be part of the Java install and this seems new enough

There is a separate ca-certificates-java package which is installed as a dependency of openjdk packages. It could be the certs in were not updated sufficiently as bullseye is no longer the current stable version. It could be a matter of reinstalling the ca-certificates-java (or dpkg-reconfigure) Otherwise it may be simpler in the long run to update the Debian distribution to the current stable version and benefit from all the recent security updates

Furthermore, in case of a problem retrieving something, Maven annoyingly saves a placeholder reminding it "this doesn't work" and it won't really try again until you do something extraordinary. This can delay your discovery that you have fixed the root-cause problem. (The placeholders are files in your ~/.m2/repository directory tree.)

by default it will re-try once a day. deps.edn does support settings to override that to always if needed

What would qualify as "something extraordinary" in this context? Deletion of the entire ~/.m2/repository tree?

you can do a more targeted deletion of course - ~/.m2/repository/org/clojure/clojure/1.11.1 for example

There is no such thing as calling a thread

thread-call means "call this function on another thread"

Exceptions as phenomena are local to a thread, they unwind the stack on the thread where they are thrown

What happens to the thread that threw through exception? Is it deleted?

Also, I used the https://github.com/reborg/parallel?tab=readme-ov-file#ppmap library to run through a list of operations with 100 concurrent threads. When an operation threw an exception, the main thread also threw. That's the origin of why I was confused about this

thread-call doesn't get used much, future is used much more often, so I don't recall the implementation of thread-call, but I believe it actually uses a thread pool under the covers, so the thread may hang around for a bit waiting to see if there is any more work to pick up before getting stopped

That is a choice the library writer made to propagate exceptions like that

If you are trying to use real threads with core.async, use clojure.core.async/thread

It is like go, but uses a real thread so it is fine for blocking operations (and you have to use the blocking channel ops)

How sure are you that the query messages are indeed equal (encodings, new line handline, escaping special characters, etc.)? You may want to sniff the communications using Wireshark and compare the outputs of your scripts.

100% sure, I simplified it to just a couple of strings so there are no lines, no special characters, etc. It's also a brand new thing, script has been running on a droplet for about 8 hours now without issues and suddenly 403s. Local REPL running queries that previously worked also 403s.

Some wild guesses: 1. Software upgrades done in the background degraded certificates. 2. A certificate has just expired and needs to be renewed.

Haven't been able to see anything useful with Wireshark but I am not exactly a power user

Here's the two requests:

(client/get url {:query-params {"p" "001:1"
                                                     "format" "files"
                                                     }
                                      :headers {"Authorization" auth}
                                      }
                                 )

r = requests.get(url,
                params= {"p": "001:1", "format": "files"},
                headers={"Authorization": auth})

Is it possible for there to be some kind of fingerprinting based on the HTTP client?

The python code doesn't have valid quoting of strings, so hard to tell if it matches

Probably worth dumping the raw requests ala curl -v in both cases and compare closely. A common mistake is to forget "Bearer " before the token auth, but that's usually responded with 401, not 403.

Given that the python code isn't valid, it seems like you aren't just copying and pasting the code you are actually running, which means you may be correcting whatever the error is when transcribing the code to share here

I would check that map keys are all at the right level

If you have been running full tilt for 8 hours it is entirely possible they decided your traffic from that ip was abusive

I checked with them for limits before I ran it, so it should be good. However, the clojure code also failed on my local machine. I ran the curl on both machines and got 200. Yea, this Python wasn't copy-pasted as I had to omit the tokens for slack, but it was in my testing

@U04CRS4B49L is there a way to get the raw request from clj-http?

The python code just isn't syntactically valid, so it is hard to tell if the query parameter names are the same

Ah, got fixed

Does the 403 actually say anything?

It's an HTML page saying I don't have access:

"\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n",

And what do you get if you make the http request from python without the auth token?

Soemthing different:

b'{"error": "User guest is not authorized to perform runapi with parameters operation=read,endpoint=search"}'

I have my local operation spit out the auth token when it's making the request just to make sure

And what do you get on the clojure side without the auth token?

It will just return something the repl will print or throw an exception which the repl will print if you type in *e

clj-http throws exceptions on not successful http codes by default but there is an option you can pass in to not do that

The same HTML page as before, meaning that the API probably didn't do an auth check against my token at all

It sure sounds like the :headers key isn't being set on the right place

That wouldn't explain why a jar that's been working for hours suddenly started throwing that error though

The fact that the clojure side and python side get different responses when not authed

Is very suspicious

I get the same HTML response when I put in this:

(client/get url {:query-params {"p" "001:1"
                                                     "format" "files"
                                                     }
                                      :headers {"Authorization" "Token abc"}
                                      })

(Token instead of Bearer is right, that's what the Python has as well)

Do you have a .netrc file?

I don't see one in my home directory

Probably the easiest way is not in clj-http itself, but you can just capture the request by swapping the URL for a dummy HTTP server running locally, that will just print it. SimpleHttpServer in Python, for example

(or even netcat)

Does the http server you are making requests give off the vibe of being home grown? I am extremely hesitant to suggest it, but maybe try a lower case A for the authorization header 😔

it's quite home-grown, it's run by the UN

(That wasn't sarcastic at you but at the UN)

127.0.0.1 - - [27/Dec/2023 15:34:09] "GET / HTTP/1.1" 200 -
ERROR:root:Host: 127.0.0.1:8000
User-Agent: curl/8.1.2
Accept: */*
Authorization: Token <my-token>


127.0.0.1 - - [27/Dec/2023 15:34:35] "GET /?p=001:1&format=files HTTP/1.1" 200 -
ERROR:root:Connection: close
Authorization: Token <my-token>
accept-encoding: gzip, deflate
Host: 127.0.0.1:8000
User-Agent: Apache-HttpClient/4.5.13 (Java/21.0.1)


127.0.0.1 - - [27/Dec/2023 15:34:46] "GET /?p=001%3A1&format=files HTTP/1.1" 200 -
ERROR:root:Host: 127.0.0.1:8000
User-Agent: python-requests/2.31.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Token <my-token>

The code running the server has been lifted without change from https://gist.github.com/phrawzty/62540f146ee5e74ea1ab

Ah, you need to url encode the query parameters

I would expect that passing them in the correct slot :query-params would automatically encode them, no? Also, still unsure about how it very suddenly stopped working? Is it possible that a server update made that happen?

Because it can't be the client library, I got an uberjar that's been working for hours and then stopped working

Here's the code that calls the local simple HTTP server:

(client/get local-url {:query-params {"p" "001:1"
                                                     "format" "files"
                                                     }
                                      :headers {"Authorization" auth}
                                      })

If you are just going be the difference in the python request above and the clojure one that is it

: is not required to be percent encoded in query parameters but it can be

So the python code and the clojure code(likely just calling some java code) may just have differing interpretations of that

I encoded the query string to look like this in the simple server:

127.0.0.1 - - [27/Dec/2023 15:47:36] "GET /?p=001%3A1&format=files HTTP/1.1" 200 -
ERROR:root:Connection: close
Authorization: Token <my-token>
accept-encoding: gzip, deflate
Host: 127.0.0.1:8000
User-Agent: Apache-HttpClient/4.5.13 (Java/21.0.1)

Wild theory, is it possible for someone to read "user agent" as an identifier of the user and to gate them from an API based on that header, maybe thinking it's a bot

Sure

Is there a universe where that makes sense, or is it always a mistake on the part of the person that would do that

And is it possible to set the user-agent manually

It is just another header

You can set it to whatever you want

Yea that was it

I set it to curl and sailed right past

Amazing

If you feel malicious you can set to a random uuid every request

thanks both, I learned a lot. Is there a universe where a server deciding to block someone on this header makes sense? To me it appears nonsensical because of how easy it is to bypass?

I literally set it to Potato just now

I have a love/hate relationship with spending a long time debugging something purely under the illusion of there's no way the right solution is this dumb...

It is more annoying that they would block the default for the java apache client and not for the python lib

It looks like a very naïve attempt at rate limiting, yes

Probably set up by some high school student

They might have some automated system that blocks high traffic user agents

I literally asked them for a rate limit and they said they didn't think there was one, although they seem to think that a rate limit was a defect in the way they talked about it 🤷 I have a lot of work I need from their system but I was happy to do it on their terms, but if there are no terms (and the only thing I got from my initial testing was a 5 minute gateway timeout) I wanted to do it as fast as possible

It is very easy to bypass, but some people consider it good etiquette to set a custom user agent if you are going to be making lots of requests

Does this mean all Apache clients on this version are blocked unless they manually set their user agents?

Crawlers for search engines typically set very specific user agents for example

It may still be a combination of user agent and source subnet

Depends how they do the block

That's good to know, although I did notify them in advance that I will want to and asked if there's anything I should do

Yeah, could be blanket user agent, could be user agent + other info

Depending on the structure of the team the blocking behavior may even belong to someone else

If you are taking to the devs of code serving whatever, it could be an ops team running a proxy in the middle somewhere that is blocking traffic they think is suspicious

I get web crawlers because they're protecting a public resource, but this is auth-protected. Why not disable the token?

New jar is deployed and running in prod, thanks!

You can use a shorter prefix such as grocery-manager, instead of the full namespace. So both in the back end and the front end you would be typing eg. :grocery-manager/type.

But I have to ask, what is at about the front end that makes it impossible to use ::type or ::gm/type?

it could be a lack of knowledge. ::gm/type works in backend code where I'm requiring [pigeon-scoops.components.grocery-manager :as gm] at the top. I don't think I can import the namespace from clj in cljs?

There is an :as-alias option to the require that should help you out 🙂 Example

(ns front-end-ns
  (:require [pigeon-scoops.components.grocery-manager :as-alias gm]))

::gm/type ;; => :pigeon-scoops.components.grocery-manager/type

Learn something new every day

thanks!

I think using keyword namespaces that are the same as code namespaces is not very good

Keywords become part of the definition of data that is passed between programs and can be stored externally out living the process that creates it

hmm that's a good point

Defining specs in a .cljc (Clojure common) file would seem a useful approach when using Clojure and ClojureScript in the same project.