clj-http 2024-01-18 | Slack Archive

Andrei Stan13:01:35

hello guys, • i want to make a clj-http.client/get request to some url, and extract the "cookie-store" • parse the response body with Jsoup and extract _csrf_token • then make a clj-http.client/post request to url/parts with {:form-params "form-params that contain the csrftoken from the first request , :cookie-store "cookie-store from first request"} and then hope that all is good and receive the data Object from response. i receive error:

; Execution error (ExceptionInfo) at slingshot.support/stack-trace (support.clj:201).
; clj-http: status 401

Looking into trace logs, seems that the connection is closing somehow i tried to make a Persistent Connection with clj-http https://github.com/dakrone/clj-http#persistent-connections should i use with-async-connection-pool`` ?? wdyt?

Janet A. Carr14:01:30

Typically session data is stored in the cookies. Are you setting up the request correctly? It doesn’t look like a connection issue to me, rather just an unauthorized request. (Status 401)

Andrei Stan14:01:00

yep, unauthorized request because i think the PHPSESSID in the cookies from the GET (r1) request is no longer available in the POST request (r2), the first request is closing, ending session

Andrei Stan14:01:20

i just need to make a persistent connection, keep alive

Janet A. Carr14:01:40

That worked?

Andrei Stan14:01:58

nope. if i copy-as cURL the request from the browser, convert that code into clojure (hardcoded one POST request), it works but when i do two requests: first to get the cookies and the csrf code second with POST, same cookies, with form-params including csrf__token, it gives me the 401 error

Janet A. Carr14:01:41

Yeah, based on what I know about sessions (and your curl request), the session is probably not ending, since the hard coded POST request would be a different connection. You could tack on some -vvvv to your cURL request for more info. Definitely sounds like some 'lost in translation' somewhere. Does your GET clj-http request have all the necessary headers? I

Janet A. Carr14:01:17

also Connection: keep-alive header lol

Janet A. Carr14:01:30

The reason I'm asking is because it sounds like you're trying to impersonate a browser, and servers can check for wonky stuff they'd only expect from a browser. Unlike an API.

Andrei Stan14:01:56

yeah, i tried with same headers, also keep-alive. somewhere the connection is discarded

11:38:15.421 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-47: Close connection
11:38:15.421 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection discarded
11:38:15.421 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.conn.BasicHttpClientConnectionManager - Releasing connection [Not bound]
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.conn.BasicHttpClientConnectionManager - Get connection for route {s}->
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-48: set socket timeout to 0
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {s}->
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to 
11:38:15.429 [nREPL-session-f1989674-b84c-4f1e-81d0-503bc7538d14] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Connecting socket to  with timeout 0

Andrei Stan14:01:45

and then opens up another connection, which will require other PHPSESSID, csrf token..

Andrei Stan14:01:16

this code works fine:

(clj-http.client/post "url/parts"
                      {:headers {"Cookie" "PHPSESSID=1nsr5286qao7dtnp7gfh6kouav;"}
                       :form-params {:page "1"
                                     :limit "3"
                                     :availableFilterOptionIds "false"
                                     :template "row"
                                     :_csrf_token "1ab73495722847007b1e5c056130f205f65e623a"}})

but when i dynamically change PHPSESSID and csrftoken (scrapped from first request) it fails with 401

2024-01-18

Channels