Clojurians Log v2

Thanks again, I have created a issue for the community to discuss: https://github.com/ring-clojure/ring/issues/522

whats the route definition?

are you using malli or something else for coercion?

I am using spec, I will get the route definition now

(s/def ::query-ref-4 (s/keys :opt-un [::agencyids
                                      ::ids
                                      ::versions
                                      ::relations]))

["/datasets"
 {:get {:tags ["Data sets"]
        :summary "Get data sets"
        :parameters {:query ::spec/query-ref-4}
        :handler (partial dataset/read-data-sets connection)}}]

(s/def ::ids (s/or :agencyids ::ids-type
                   :all ::sdmx/all))

(s/def ::ids-type (s/and string? #(re-matches comma-sep-ids-regex %)))

(def comma-sep-ids-regex (re-pattern (str "^(" sdmx/id-regex ")?(," sdmx/id-regex ")*$")))

(def id-regex #"[A-Z0-9](_?[A-Z0-9]+)*")

(s/def ::all (partial = "all"))

Do you think its a regex stuff up? If it is I don't understand why it is dropped when parsing the query string?

:data {:coercion reitit.coercion.spec/coercion
              :muuntaja muuntaja-sdmx-codera
              :middleware [parameters/parameters-middleware
                           muuntaja/format-negotiate-middleware
                           muuntaja/format-response-middleware
                           custom-exception-middleware
                           muuntaja/format-request-middleware
                           coercion/coerce-response-middleware
                           coercion/coerce-request-middleware
                           multipart/multipart-middleware]}

so an optional key gets dropped if it's value doesn't match the schema, hmm

that might just be how spec-tools coercion works, let me check

Thanks, much appreciated

no, I can't get the same behaviour just by playing around with st/coerce and st/conform

let me try to reproduce this more directly

I can also try reproduce a complete example if that would be useful?

yeah, that would help

Perfect, will post when I have something

spec coercion uses strip-extra-keys-transformer by default: https://github.com/metosin/reitit/blob/master/modules/reitit-spec/src/reitit/coercion/spec.cljc#L25-L26

ah but that is only for body params

Here is a small example https://github.com/byrongibby/reitit-test

Good morning @juhoteperi, to confirm, the observed behaviour in this case is the expected behaviour?

But this isn't an extra key in this case, it's specified in ::query-ref-4. Thanks for the example Byron, I'm looking into it now.

Thanks!

Yeah, I can reproduce the problem. Interesting! With

?agencyids=MY_AGENCY&ids=id_1 

?agencyids=MY_AGENCY&ids=%3c%%3d77%2a77%%3e

Yep, it is strange, I wonder if this is what is intended

Thanks for reproducing!

I think this is caused by some earlier stage, not reitit coercion. Reitit coercion reads :query-params and writes into :parameters :query

Makes sense

and :query-params is missing the weird :ids even when I comment out

;;:coercion reitit.coercion.spec/coercion

reproduced:

user=> (require '[ring.middleware.params :as p])
nil
user=> (p/assoc-query-params {:query-string "agencyids=MY_AGENCY&ids=ID_1,ID_2"} "UTF-8")
{:query-string "agencyids=MY_AGENCY&ids=ID_1,ID_2", :query-params {"agencyids" "MY_AGENCY", "ids" "ID_1,ID_2"}, :params {"agencyids" "MY_AGENCY", "ids" "ID_1,ID_2"}}
user=> (p/assoc-query-params {:query-string "agencyids=MY_AGENCY&ids=%3c%%3d77%2a77%%3e"} "UTF-8")
{:query-string "agencyids=MY_AGENCY&ids=%3c%%3d77%2a77%%3e", :query-params {"agencyids" "MY_AGENCY"}, :params {"agencyids" "MY_AGENCY"}}

could the problem be that the garbage is not valid UTF-8 so it's being discarded?

perhaps this line in ring.util.codec is discarding the value: https://github.com/ring-clojure/ring-codec/blob/master/src/ring/util/codec.clj#L148

In your opinion this is not the behaviour that one would want right?

user=> (java.net.URLDecoder/decode "agencyids=MY_AGENCY&ids=%3c%%3d77%2a77%%3e" (java.nio.charset.Charset/forName "UTF-8"))
Execution error (IllegalArgumentException) at java.net.URLDecoder/decode (URLDecoder.java:237).
URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 0 in: "%3"

that's the exception that gets swallowed

Aha

it's hard to say what's the right behaviour here, but yeah, I think I would prefer the params-middleware to return a HTTP 400 Bad Request in cases like this

you could consider filing a bug against ring

Thanks a lot for you help with this, I will think about it

bsless (the maintainer) might have some opinions on this

my message from above is a good repro for the bug: https://clojurians.slack.com/archives/C7YF1SBT3/p1740726284013619?thread_ts=1740666397.391129&cid=C7YF1SBT3

Great, thanks

Did you encounter this with real use (ring-jetty-adapter, http-kit or such) or only when calling the handler function directly?

And do http clients allow making such a request?

the data looked to me like it was generated by a security audit tool

@joel.kaasinen is exactly right, this issue came up during a penetration test. I can confirm that Firefox allows this request

The backend uses ring-jetty-adapter