Fork me on GitHub
#reitit
<
2021-10-29
>
ikitommi12:10:51

@viebel yes it's possible. Please read the coercion docs. If not there, there should be tests to cover that. Doc PRs welcome

ikitommi12:10:39

Also, there is an issue about inlining meta-merge to give easier support for this

Kira McLean15:10:02

Hello! I’m wondering if anyone has any examples they could share of custom-coercing query params in reitit (I’m using malli coercion in a reitit-ring app, but examples using libs other than malli would be helpful too). I want to convert a query param on a single endpoint from a comma-separate string of numbers to a tuple of 4 floats and having a harder time than I expected.

Kira McLean15:10:13

At first I tried to use a middleware just on that endpoint that would transform the query param, but it appears that parameter coercion happens before middlewares run, so of course my custom middleware never executes because the coercion step fails first.

Kira McLean15:10:47

I also tried adding a :reitit.middleware/transform key (https://cljdoc.org/d/metosin/reitit/0.5.15/doc/ring/transforming-middleware-chain) to try to re-arrange the order of the middlewares for just that endpoint, but either it doesn’t work like I thought or I did it wrong

Kira McLean15:10:24

So now I think I need to construct a custom coercer with my own transformer and looking for examples/advice 🙂

dharrigan17:10:51

I had a requirement like that, ages ago. I felt it was just easier to accept in the comma-separated list, then have a little mapper that would look for that parameter and split it into how I want it to be.

dharrigan17:10:19

i.e., :status (if-not (blank? status) (split status #",") ["DRAFT" "ONGOING"])

dharrigan17:10:27

Where status was a query param coming in

dharrigan17:10:50

[:status {:optional true} string?] ; can query on "DRAFT,ONGOING" for example...

Ben Sless17:10:49

I did something very similar a few days ago with very custom coercion, had to create a custom instance and specify transformers for everything, but it worked out

Ben Sless17:10:00

I can lay out a more detailed example tomorrow

Ben Sless17:10:33

the solution is a big ugly

Kira McLean18:10:49

I think I’ve got something nearly working in a similar way.. but yeah very hairy. I have a custom coercer accepting a custom provider which uses a custom transformer that has my own custom string decoder for the one param. I’m mostly just guessing how malli internals work at this point, which obviously is a frustrating way to code! I was thinking it would be ideal to just figure out how to do this with the existing coercion/transformation stuff so that in the handler I just have nice params I know are valid, but certainly just doing it manually at a later stage is an option, and maybe the way to go in this case if this pile of providers/transformers/decoders/schemas/registries is really the answer.. I’m not at all confident it is, though

juhoteperi19:10:36

@kiraemclean Use decide/string to control how value is converted from string to collection or whatever: https://github.com/metosin/malli#value-transformation. [:map [:status [:tuple {:decode/string split-and-coerce} :double :double :double :double]]]

Kira McLean19:10:54

what is split-and-coerce in this example? just a function that takes the incoming param?

juhoteperi19:10:53

Yeah, something like (fn [s] (map #(Double/parseDouble %) (str/split s #",")))

juhoteperi19:10:52

In fact you might not even need parseDouble, built-in :double decoder might work

Kira McLean19:10:06

When I do something like that I get an invalid-schema error from malli.. it appears I can mitigate it by naming that function and adding it in the :registry under :options in a custom coercer (`reitit.concern.malli/create`), but it doesn’t ultimately fix the issue

juhoteperi19:10:10

(def schema
  [:tuple {:decode/string (fn [s] (str/split s #","))}
   :double
   :double
   :double
   :double])

(m/decode schema "1,2,3,4" mt/string-transformer) => [1.0 2.0 3.0 4.0]

juhoteperi19:10:58

split is enough, because tuple decode/string happens before :double decode, and built-in decode fn for :double is able to parse those separate numbers

Kira McLean19:10:24

this much makes sense but I’m still lost about how to integrate into an app using malli to coerce ring query params.. I’m not calling decode anywhere explicitly, and I’m still quite confused about how the transformers/decoders are ultimately threaded through to the router,

juhoteperi20:10:51

:coercion reitit.coercion.malli/coercion is enough for this case

Kira McLean20:10:37

🙈 I’m definitely missing something.. so when I have a ring/router that includes :coercion reitit.coercion.malli/coercion and a route defined something like this:

["/path/"
    {:get
     {:handler ,,,handler here,,,
      :parameters  {:query [:map
                            [:my-param {:optional true :decode/string my-decode-fn}]]}}}]
and try to run my app, I get the error :malli.core/invalid-schema

Kira McLean20:10:16

I think I’ve convinced myself I need to go down the path Ben was describing, but it sounds like he may have figured out a way that’s slightly less brutal than the initial one I found..

juhoteperi20:10:33

In this example, the [:my-params] map key is missing the schema

Kira McLean20:10:47

the full message is :malli.core/invalid-schema {:schema {:optional true, :decode/string my-decode-fn}}"

juhoteperi20:10:37

["/path/"
    {:get
     {:handler ,,,handler here,,,
      :parameters  {:query [:map
                            [:my-param {:optional true :decode/string my-decode-fn} :string]]}}}]

juhoteperi20:10:25

If a vector under :map schema has two elements, the second one is the schema for that property. If it has three elements, second is the options map and third the schema.

Kira McLean20:10:04

ah ok so I need to add :string.. trying…

juhoteperi20:10:32

Well what ever is the schema for the type you want to decode into

juhoteperi20:10:54

could be [:tuple :double :double] also if that's what you want to decode into

juhoteperi20:10:03

And the :decode/string option could be at the property schema instead of at the map key-value schema.

Kira McLean20:10:19

ohh that’s progress.. now the validation is failing, saying “should be a string”.. which I guess is fair, at this point it’s not a string, it’s my tuple of 4 numbers, but it is actually using my decoder!

juhoteperi20:10:56

["/path/"
 {:get
  {:handler handler-fn
   :parameters  {:query [:map
                         [:my-param {:optional true :decode/string my-decode-fn} [:tuple :double :double]]]}}}]

["/path/"
 {:get
  {:handler handler-fn
   :parameters  {:query [:map
                         [:my-param {:optional true} [:tuple {:decode/string my-decode-fn} :double :double]]]}}}]
Both of these work, but I think the second one makes more sense

Kira McLean20:10:33

so when I run it now the validation fails with “invalid type”.. this is further than I’ve gotten yet, though.. appreciate you explaining it all for me very much!

Kira McLean20:10:53

and this is certainly simpler than what I was doing before, and gets at least as far..

Kira McLean20:10:18

oooook it’s working! I can get a hold of the parsed/coerced params in my handler now. this is amazing, and so much better than what I was doing.. thank you so much @U061V0GG2. I owe you one if we ever cross paths 😄

🎉 1
Kira McLean20:10:24

this is so amazing and clean just like I was dreaming of. thanks so much.

👌 1
juhoteperi19:10:36

No reason to setup custom transformer for this, built-in string-transformer will look at this schema property so you can control it per value.

Ben Sless19:10:25

Unless you're trying to do custom decoding (which I was) and it required dragging it kicking and screaming all the way up to the coercer

ikitommi19:10:50

Ideas most welcome how to make simpler.

Kira McLean19:10:41

cool! I’m just getting into malli for the first time now. it’s an amazing library, thanks for making it 🙂 once I figure this out it might be more realistic to be able to contribute.

juhoteperi19:10:44

@ben.sless What do you mean "custom decoding"? Like different type of input than just strings? Or need to decode same schema different way in different cases?

Ben Sless19:10:08

as in :decode/my-terrible-use-case

juhoteperi19:10:36

Okay, there are valid cases for having own transformer and own decoding property

Ben Sless19:10:02

that certainly was one of them. If you're interested in a detailed usage report I can run it by my employer and even share the ugly bits of code, otherwise I can try to sketch things in general lines and mock examples

Kira McLean19:10:23

do you mean you were decoding something other than a string, or you wanted to process the string in some weird custom way?

Ben Sless19:10:48

For example, I was decoding an int, but also wanted to divide it by 1000

Ben Sless19:10:16

so I was counting on a string decoder for :int, then added :decode/round {:leave ,,,}

Kira McLean19:10:28

right.. I kind of want to do something similar. I want to parse an incoming query param that looks like "1,2,3,4" into [1 2 3 4]

Ben Sless19:10:54

Provide a (mt/transformer {:name :round}), then you have a transformer

Ben Sless19:10:07

Right, so you want it to happen before

Ben Sless19:10:25

I'm assuming the number of elements is unbounded

Kira McLean19:10:17

that’s the confusing part for me since I’m using this to try to parse query params in the middle of a ring/reitit app.. I made some progress building a custom coercer with a custom string transformer provider.. but that quickly got out of hand and ultimately still didn’t work

Kira McLean19:10:31

no specifically 4, it’s coordinates for a bounding box

Ben Sless19:10:54

so you can do something like [:tuple {:decode/query-params #(str/split #"," %)} :int :int :int :int] Then provide a transformer which works in order, i.e. (mt/transformer {:name :query-params} mt/string-transformer)

Ben Sless19:10:47

then use that transformer when instantiating the coercion in reitit for query-params

Ben Sless19:10:09

btw, @U061V0GG2 might be worth making -provider public

Ben Sless19:10:49

We have a very similar use case, I also needed to aggressively massage query-params

Kira McLean19:10:09

where do you provide that transformer? maybe some code would be helpful.. I have tried e.g.

(mcoerce/create
 {:transformers {:body {:default mcoerce/default-transformer-provider
                        :formats {"application/json" mcoerce/json-transformer-provider}}
                 :string {:default <CAN PROVIDE CUSTOM TRANSFORMER PROVIDER HERE>
                          }
                 :response {:default mcoerce/default-transformer-provider}}
  :error-keys #{:type :coercion :in :schema :value :errors :humanized :transformed} ;; set of keys to include in error messages
  :compile mu/closed-schema ;; schema identity function (default: close all map schemas)
  :validate true ;; validate request & response
  :enabled true ;; top-level short-circuit to disable request & response coercion
  :strip-extra-keys true ;; strip-extra-keys (effects only predefined transformers)
  :default-values true ;; add/set default values
  :options nil ;; {:registry swirrl-custom-registry} ;; malli options
  })

Ben Sless19:10:12

[:transformers :query-params], start there

Kira McLean19:10:28

where I made a custom transformer provider to use, but it quickly unravelled all the way to some very nitty gritty details that I don’t yet fully understand (and hasn’t worked yet), so I was wondering if there was some way to do it I’m missing

Kira McLean19:10:09

so leave the :string one as the default, but add a query-params key with my custom one?

Ben Sless19:10:32

This touches on a question I asked previously in a channel, but if adding a transformer in query-params doesn't solve it, try adding the same transformer in string, too

Kira McLean19:10:36

did you go source diving to figure out what that should look like?

Ben Sless19:10:49

yes, didn't find answers, though, only questions

Kira McLean19:10:45

hah.. relatable

Ben Sless19:10:56

Eventually I managed to build a very minimal example and just tested out the combinations until I found what worked

Ben Sless19:10:38

don't despair, it's a solvable problem, and lucky for you I already banged my head against that wall

Ben Sless19:10:59

I'll build some minimal example tomorrow to illustrate a possible solution

Ben Sless19:10:06

now is sleepy time

Kira McLean19:10:54

I’m wondering if it’s worth doing malli coercion on the params in the first place.. it’s not a hard problem to deal with manually, would just be cooler if it was all magically done by the router

Kira McLean19:10:21

cool, no worries either way, but yeah if there’s some example you’re allowed to share seems like that would be super useful. have a great night! and thanks for this much help!

Ben Sless19:10:07

Resounding YES to your question. Shoving all coercion and parsing to the edge cleans so much damn cruft out of your code, leaving only with business logic

Ben Sless19:10:49

you'll find that once you get the hang of it, you even get a positive ratio of eliminated code

Kira McLean19:10:12

yeah.. I definitely see the benefits. painful, though!

Ben Sless19:10:46

Just to pinpoint the friction point I came across, might be different from Kira's case, I have custom transformers with my schema (works great, besides the leave/compile magic). I wanted to use that transformation for query params, ended up having to use the same transformer for string, still not sure why