Fork me on GitHub
#yada
<
2019-01-18
>
kwladyka17:01:45

@borkdude hey, we were talk about exceptions on web page in yada before somewhere. So how do you deal with it https://clojurians.slack.com/archives/C0702A7SB/p1547229018025500 ?

kwladyka17:01:00

It is happening for me for all types of “errors”

borkdude17:01:41

this is basically what I do: https://gist.github.com/borkdude/441379ec59d8da7f72348453ee15f32a note: this code is almost 2 years old and yada might have better ways right now

kwladyka17:01:24

But is it only for definend content-type?

malcolmsparks17:01:38

You can add :show-stack-traces? true to your yada resource map

kwladyka17:01:46

I don’t see in this code option to turn off it for all errors. I see you use defmethod render-error "text/plain"

malcolmsparks17:01:48

Or of course, :show-stack-traces? false

kwladyka17:01:01

@malcolmsparks woo how can I know about that?

borkdude17:01:02

I turn off stacktraces in prod.

malcolmsparks17:01:34

all the schema is defined in yada.schema. It's a single ns that defines all possible things you can have in a yada/resource

malcolmsparks17:01:50

there should definitely be a documentation chapter explaining it all

kwladyka17:01:08

Maybe I miss something big, but I don’t see it

malcolmsparks17:01:09

but it's good to know where the actual source is because documentation can be out of date

borkdude17:01:11

that was added 23rd of May 2017, I probably wrote this code before it

malcolmsparks17:01:11

if you want to do something to multiple resources at once, the standard answer is to clojure.walk/postwalk your resources

malcolmsparks17:01:32

you can add some namespaced entries in your resources if you need to filter them, etc.

borkdude17:01:44

@malcolmsparks we have a common resource model that we merge with overridable data per resource, that also works

malcolmsparks17:01:52

yes, that too - it's data

kwladyka17:01:19

So I have to add this to all resource definition? No way to set it when start yada?

malcolmsparks17:01:48

no - everything is at a resource level - this is intentional

malcolmsparks17:01:59

sometimes you want different policies for different resources

malcolmsparks17:01:21

and yada discourages putting policies on the global or 'route prefix' level

malcolmsparks17:01:53

This is partly to make resources self-contained (which can help when testing them individually).

malcolmsparks17:01:29

It's also about making it easier to reason about resources on their own

kwladyka17:01:17

sure I understand it, but this one I would like to see as global conf 🙂 So I will ended up with function wrapper to create resource without stacktrace like everybody 😉

kwladyka17:01:25

But I understand the concept

borkdude17:01:45

@kwladyka like I said, you can make a “global conf” by using a global map with your default and then merge differences in per resource

borkdude17:01:07

@kwladyka in our global resource map there’s for example auth stuff

malcolmsparks17:01:21

I've begun to work more actively on yada lately, and I'm trying to add documentation to each feature as I commit. But I do have some outstanding documentation to author.

malcolmsparks17:01:33

One good general tip is to read the source code.

borkdude17:01:52

Also the Edge app is a good example

malcolmsparks17:01:31

Once you understand how yada works it's not that frightening. The overall design is quite simple once you understand it - it's a series of functions that are called in a sequence, acting on a single map (called the yada context)

malcolmsparks17:01:42

once the context is fully built, the response is ready to be returned

malcolmsparks17:01:17

The trick is that there is enough data in the yada context to figure out things like content negotiation, conditional responses and so on.

kwladyka17:01:21

agree, as a feedback (please think about it positive! I have good intention!) in my feeling doc is too long without right value. It could be 20% of that length with the same value or better. I mean there is a lot of content which doesn’t give me any improvement.

malcolmsparks17:01:54

thanks for the feedback

malcolmsparks17:01:16

we are working on some Edge tutorials which may be easier to follow

kwladyka17:01:50

I understand, it is in progress

borkdude17:01:41

@malcolmsparks have you thought about swapping bidi for e.g. reitit to make it work with spec?

kwladyka17:01:45

But I really like idea about yada and I am full of hope 🙂

malcolmsparks17:01:59

@borkdude I have looked at reitit and really like it - however, it does have some constraints such as segmenting on / only.

malcolmsparks17:01:22

There is some integration between bidi and yada to makes URI forming 'easier' which I dislike

borkdude17:01:00

you dislike easier?

kwladyka17:01:20

@borkdude Why reitit use [reitit.coercion.spec] instead of core one? Sorry I am asking, but I don’t know this library. Is it possible to use it with [clojure.spec.alpha :as s]?

borkdude17:01:40

I don’t know reitit at all.

malcolmsparks17:01:30

@borkdude there are alternative ways of doing bidi url forming without going via the yada ctx, and I think the use of the yada ctx is a bit of an anti-pattern. I think bidi and yada should work more independently. I don't think there's any reason today why yada and reitit can't work well together. It's not really a case of 'swapping' bidi for reitit, just to support both and give the user the choice.

malcolmsparks17:01:48

Edge could have some examples where reitit is used instead of bidi.

borkdude17:01:16

yeah. but yada also is coupled with schema right now, right?

borkdude17:01:27

you can just turn that off by not using it I guess

borkdude17:01:40

but then the schema/spec is not part of the resource anymore

malcolmsparks17:01:35

I was thinking of swapping out schema for clojure.spec when clojure.spec first came out. But I have since reconsidered. Schema plays 2 roles in yada - the first is by checking the data given to yada resources. The second is for parameters and Swagger. I don't think clojure.spec is a good choice for either.

malcolmsparks17:01:41

We could take schema out of yada, but then you'd lose the shorthands and they're quite handy. Not having coercions would make yada resources quite a bit more verbose.

borkdude17:01:56

makes sense

malcolmsparks17:01:49

I do think that the use of schema in yada parameters is a bit of a problem. I would prefer schema just be an internal thing really.

malcolmsparks17:01:11

For parameters, I think that parameter validation (to produce 400 errors) is really a separate concern and not one that yada should be involved with. The declaration of parameters isn't part of the HTTP standards and was really driven by Swagger. I used to think it was a big advantage to have Swagger 'built-in' but now I think it would be fine if all the Swagger stuff were available as an extension and not part of yada core. That would reduce the dependencies and overall size of yada a lot.

malcolmsparks17:01:43

(I still think parameters validation is important, but something where there should be more options rather than 'built-in')

kwladyka17:01:39

@malcolmsparks > I don’t think clojure.spec is a good choice for either. Why?

malcolmsparks18:01:57

2. I don't think clojure.spec is designed for validation of web forms - there are more purpose-built libraries for that

kwladyka18:01:31

Form validator based on clojure spec

kwladyka18:01:46

It seems to be fine, but maybe I miss something :thinking_face:

kwladyka18:01:32

I really want to use the same code for validation on BE and FE.

dominicm18:01:15

@kwladyka you should not run spec against data from the Internet until you have validated it.

kwladyka18:01:12

Can you expand that?

kwladyka18:01:00

I mean I use spec + fn for not standard things and it works without any issue.

kwladyka18:01:35

But the biggest advantage is I can use spec from BE on FE and be sure everything is consistent

kwladyka18:01:00

Writing different code validation on FE and BE twice sounds bad for me

kwladyka18:01:11

But if you can five me further input which will change my view it will be great!

dominicm18:01:13

It's a security risk, due to not being able to constrain the registry to only safe ones

kwladyka18:01:47

What do you mean?

kwladyka18:01:08

code example?

dominicm18:01:30

If a library contains a slow spec, for example, let's say it takes 1s per character. Then I can send a long string with that key, even though your endpoint doesn't accept it.

dominicm18:01:04

E. G. {:ring/method "longstring"}, I can use that to easily dos your site

dominicm18:01:23

Traditionally this happens with password hashing and not limiting string length

kwladyka19:01:39

mmm but that is why spec are? To check lenght of that string before hashing

kwladyka19:01:14

The same situation with Schema. App has to check it in right moment.

dominicm19:01:24

Schema use a restricted map of validations, spec uses an open map

kwladyka19:01:04

not sure what it means

dominicm18:01:28

Additionally, if you want to spec that :password and :password-repeat are equal, you cannot put the error message on :password-repeat, only on the whole form.

kwladyka18:01:39

It can be done, because you can recognise spec name (s/def ::foo/bar ...)

kwladyka18:01:48

But fn for that purpose are probably better

dominicm18:01:52

I don't understand. How does this work?

kwladyka18:01:47

While you can get reason of fail like [::form ::form-map ::password ::password-not-empty] you can get [::form ::form-map ::password-repeat ::password-not-empty] instead

kwladyka18:01:03

so you know error is about password-repeat

kwladyka18:01:32

all is about spec namig

kwladyka18:01:44

:via spec key is the point of get this info

dominicm19:01:02

But my spec is on ::form

kwladyka19:01:38

(ns form-validator-doc.spec
  (:require [cljs.spec.alpha :as s]
            [clojure.test.check.generators]))

(s/def ::email (s/and string? (partial re-matches #"^[a-zA-Z0-9._%+-][email protected][a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")))

(s/def ::password-not-empty not-empty)
(s/def ::password-length #(<= 6 (count %)))
(s/def ::password (s/and string? ::password-not-empty ::password-length))

(s/def ::form (s/keys :req-un [::email ::password]
                      :opt-un [::password-repeat]))

kwladyka19:01:25

if you create (s/def ::password-repeat ...) you can get for example [::form ::form-map ::password-repeat ::password-not-empty]

dominicm19:01:29

But you are not validating that the fields are equal

kwladyka19:01:43

it can be done in spec!

kwladyka19:01:54

but i prefer fn

dominicm19:01:57

But the :via will not point at the right field

kwladyka19:01:15

yes, then it is more complex. That is why I use spec + fn

kwladyka19:01:36

Most of the validation can be done by spec

dominicm19:01:40

But sure, you can do anything if you add extra validation, but then you have to put custom logic on top

dominicm19:01:09

You lose the benefits of spec

kwladyka19:01:00

In that specific example not, because I don’t even want to have :password-repeat as part of my deep logic. It is not something what I will send to BE. It is only for UI purpose.

kwladyka19:01:35

How would you do it better? spec + Schema? It sounds like doing work twice.

kwladyka19:01:47

and issues about consistency

kwladyka19:01:56

But I would love to know how you are doing that 🙂

dominicm19:01:11

I like vlad for forms

dominicm19:01:19

The messages are good by default

kwladyka19:01:16

lib looks like not maintained anymore

kwladyka19:01:52

Anyway it is very interesting discussion! 🙂

dominicm19:01:58

It's a finished library pretty much

kwladyka18:01:26

But I agree spec needs support of fn for coercion

kwladyka18:01:25

But in practice you don’t send :password-repeat to BE, it is checking only on FE

kwladyka18:01:10

So you don’t want to have :passowrd-repeat in main spec anyway

kwladyka18:01:45

Or check things like “this username already exist”

kwladyka18:01:35

@malcolmsparks {:show-stack-traces? false} how can I add this to for example HTTP 500? Do I have to define all HTTP responses for that purpose?

kwladyka18:01:49

ok this was bad example probably

kwladyka18:01:11

because 500 is about resources itself

kwladyka18:01:08

Oh I see, if yada doesn’t find any resources it returns 204 without stacktrace

kwladyka18:01:50

ok, never mind, I see how it works 👍

kwladyka19:01:55

I have to say after second try of yada I like it even more

👍 5
kwladyka19:01:02

interesting fact: after change liberator to yada I removed about half dependencies and about half code

kwladyka19:01:19

*with the same effect

borkdude19:01:32

I had the same experience with an http-kit/ring based stack. I really like the batteries included of yada. I don’t mind the dependencies as long as it’s cohesive

kwladyka19:01:16

Do you know best practice example for yada + lacinia? Something what can inspire me or boost my process of learning how to glue things

borkdude19:01:46

lacinia does have a lib for pedestal. I’m not sure what exactly it’s doing and I never used it in anger

kwladyka19:01:08

it is just graphql library

borkdude19:01:12

I did try lacinia once I must say, but we dropped it because the GraphQL standard was too narrow for us

kwladyka19:01:46

My goal is to make GraphQL API with option to login by web browser + by token for integrations

kwladyka19:01:55

There is a lot of decisions to make

borkdude19:01:16

maybe Edge uses GraphQL

kwladyka19:01:45

Generally when join to project this part is already done. But now I start something from 0 and I have to decide about everything. It is biggest challenge, than I thought 🙂

borkdude19:01:07

the example I just posted uses lacinia + yada

kwladyka19:01:32

Maybe I am not sure what is the best way for auth

borkdude19:01:06

maybe auth should have its own endpoint?

kwladyka19:01:09

I have no good idea how to do it using only GraphQL in secure way. I need this POST REST API method to login (get token based on email and password)

kwladyka19:01:47

Maybe it should be like that or maybe it should be moved to GraphQL mutations, but then it start to confuse me how to implement it

kwladyka19:01:04

yes, this is exactly how I am doing it now

kwladyka19:01:22

but there is not a lot of people with who I can do brainstorm about that topic

kwladyka19:01:51

actually I don’t know people personally with deep experience about it I think

kwladyka19:01:18

system have REST + GraphQL, because they exited before etc.

borkdude19:01:21

maybe there’s a #graphql channel

kwladyka19:01:33

yes, I know that channel

kwladyka19:01:14

but there is not a lot of system which based only on GraphQL without REST API I think, so people don’t have this experience

kwladyka19:01:23

including me 😛

borkdude19:01:47

Me neither 😉

kwladyka19:01:51

And there are some resources like download PDF which are not GraphQL… ok it is off topic now 🙂

borkdude19:01:51

GraphQL is designed for fetching nested data, to prevent unnecessary data sent from the sever, right? What stops you from making extra REST points together with it, that do not have that problem?

kwladyka19:01:53

Do you know any place to talk about architecture of systems in that context?

kwladyka19:01:03

It is something deeper than only GraphQL

kwladyka19:01:31

yes, it could be done, but still there is a lot of decision to make. Like for example about auth.

kwladyka19:01:39

It is about decisions 🙂

kwladyka19:01:40

For example I started with sessions in Redis, but now I change concept to not use sessions at all

kwladyka19:01:20

instead I want to send token in each API call

kwladyka20:01:21

How to run yada to use easy parameters from POST? curl -vs -H "Accept: application/json" --data 'query="{ game_by_id(id: "1237") { name designers { name }}}"' -X POST -i <- for example I want to read query parameter form POST method.

kwladyka20:01:38

Should I use ring middlewares?

stijn22:01:02

@kwladyka I can give you an example of yada + lacinia when I'm at the computer

stijn22:01:10

We do authentication in http (session / token) and authorization in lacinia obviously

stijn22:01:19

This might help you decide

kwladyka22:01:26

Thanks, example of code will definitely boost me

stijn22:01:27

It's just much more convenient to do authentication outside of graphql because of all the web standards like cookies

stijn22:01:19

But that also means you need at least one non graphql endpoint for login e.g.

stijn22:01:38

Gotta go, remind me later if I forget 😊

👍 5
kwladyka22:01:03

Different topic: how do you make POST request (yada/response-for authentication :post "/authentication" {:body "{\"email\": \"\", \"password\": \"qwaszx\"}"}) ? I am trying to use response-for but I can’t guess the right way 🙂

kwladyka22:01:18

{:status 400,
 :headers {"content-length" "67", "content-type" "text/plain;charset=utf-8"},
 :body "No body present but body is expected for request.\r\n\r\n{:status 400}\n"}

kwladyka22:01:46

curl --data '[email protected]&password=qwaszx' -X POST -i                         Fri Jan 18 23:08:35 2019
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 22:09:11 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}
Also this issue. I assume I have to have ring middleware to make :parameteres work? But on the other hand it will be odd to add extra ring middleware on default yada things.

kwladyka22:01:25

Seriously I have no idea how to use POST in yada after a few hours of trying 😕 I will appreciate example of code.

kwladyka22:01:53

And for some reason it doesn’t work also with curl

:parameters {:form {:email String
                                 :password String}}
I was trying with :form and :body. It just doesn’ work.
{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

kwladyka22:01:55

Is it possible to use peridot with yada? How?

borkdude22:01:53

@kwladyka I have this in my tests:

(defn request [method url clj-body]
  {:method method
   :content-type "application/json"
   :accept "application/json"
   :as :json
   :url url
   :body (cheshire/encode clj-body)
   :throw-exceptions false})

(defmacro with-server
  "Runs resource in server and defines url for use in body"
  [url-bind resource & body]
  `(with-level :level/off "yada.handler"
     (let [resource# ~resource
           vmodel# (vhosts-model [:* ["/api/foo" resource#]])
           listener# (y/listener vmodel#)
           port# (:port listener#)
           close# (:close listener#)
           ~url-bind (str "" port# "/api/foo")]
       [email protected]
       (close#))))

(deftest schema-validation-error-test
  (er/define-error-renderers true)
  (with-server url
    (y/resource {:methods
                 {:post {:parameters {:body {:foo s/Str}}
                         :response (fn [ctx] {:a 1})
                         :produces #{"application/json"}
                         :consumes #{"application/json"}}}})
    (let [response (client/request
                    (request :post url {}))
          body (cheshire/decode (str (:body response)) true)]
      (is (= "missing-required-key" (-> body :error :foo)))
      (is (= "Schema validation error" (-> body :message))))))

kwladyka22:01:06

Hmm I will try to translate it to (response-for ... ) yada fn

kwladyka22:01:54

It looks like yada doesn’t see parameters

borkdude22:01:58

I don’t use response-for, because it has some limitations, I can’t remember which, maybe to do with not all interceptors being present

borkdude23:01:41

it’s been 2 years since I wrote this code. This code only tests some error logic. Most of my other API tests I run completely outside of yada, as integration tests

borkdude23:01:32

the with-server macro starts a real aleph server, so it’s more like what you would normally see

kwladyka23:01:49

Ok I have to do small steps. Any idea why yada doesn’t see POST parameters at all?

kwladyka23:01:38

(def authentication
  (build-resource
    {:id :authentication
     :consumes #{"application/x-www-form-urlencoded"}
     ; :access-control
     :methods {:post {:produces "application/json"
                      :parameters {:form {:email String
                                          :password String}}
                      :response (fn [{:keys [parameters] :as ctx}]
                                  (println (pr-str ctx))
                                  (println (pr-str parameters))
                                  ;(println (pr-str form))
                                  (let [{:keys [email password]} (:body parameters)]
                                    {:token "foo"
                                     :email email
                                     :password password}))}}}))
curl -d '[email protected]&password=qwaszx' -X POST -i 
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:11:04 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}

borkdude23:01:34

yeah, you need get rid of the :form key in your schema

borkdude23:01:04

I think… let me check

kwladyka23:01:05

what should be there instead?

kwladyka23:01:39

But the point is (println (pr-str ctx)) doesn’t show me any parameters

kwladyka23:01:58

So something is wrong

borkdude23:01:02

I have {:parameters {:body …}}

kwladyka23:01:09

I was trying with that too

borkdude23:01:41

oh I see you’re dealing with a form, let me check if I have such an example

kwladyka23:01:58

Do you do something extra to let yada see POST parameters in request?

kwladyka23:01:45

It doesn’t work for me with any example. I don’t have parameters in`ctx`

borkdude23:01:29

can you try with a different consumes thing

borkdude23:01:47

e.g. application/json and then post a JSON body

kwladyka23:01:50

:parameters is {}

kwladyka23:01:30

the same with JSON

kwladyka23:01:41

:parameters is {}

kwladyka23:01:14

ok but :body should be ok now, let me check

kwladyka23:01:55

ok it works good for JSON

kwladyka23:01:17

but for POST application/x-www-form-urlencoded so typical form it doesn’t work

kwladyka23:01:33

oh I see I have to have parameters in map to have them in ctx. So for POST it has to be different way than :form and :body hmm

kwladyka23:01:05

But I have no idea how to set it then

kwladyka23:01:47

hmm my guess is yada doesn’t know how to handle application/x-www-form-urlencoded

kwladyka23:01:37

this is super weird:

curl -H "Content-Type: application/x-www-form-urlencoded" -X POST -i                   Sat Jan 19 00:35:45 2019
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 45
Content-Type: application/json
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:36:06 GMT

{"token":"foo","email":null,"password":null}
curl -d '[email protected]&password=qwaszx' -X POST -i                             Sat Jan 19 00:36:06 2019
HTTP/1.1 400 Bad Request
Content-Length: 129
Content-Type: text/plain;charset=utf-8
Server: Aleph/0.4.4
Connection: Keep-Alive
Date: Fri, 18 Jan 2019 23:36:56 GMT

Bad form fields

{:status 400,
 :error
 {:email missing-required-key,
  :password missing-required-key,
  nil disallowed-key}}
So when I send empty POST I don’t have any error, but when I send this fields I have error these keys are required

borkdude23:01:48

it could be that yada doesn’t coerce the body against the schema in that content-type

kwladyka23:01:02

for this setting

{:form {:email String
                                          :password String}}

borkdude23:01:07

is that documented somewhere?

kwladyka23:01:15

What exactly?

borkdude23:01:24

form urlencoded post body

kwladyka23:01:45

no but there are examples in the doc about POST

kwladyka23:01:51

so it should work

kwladyka23:01:35

Looking on this test I don’t understand why it doesn’t work for me

borkdude23:01:38

also, I can’t tell what your build-resource function returns. I’m off to bed. good luck 🙂

kwladyka23:01:01

#yada.resource.Resource{:id :authentication,
                        :consumes [{:media-type #yada.media_type.MediaTypeMap{:name "application/x-www-form-urlencoded",
                                                                              :type "application",
                                                                              :subtype "x-www-form-urlencoded",
                                                                              :parameters {},
                                                                              :quality 1.0}}
                                   {:media-type #yada.media_type.MediaTypeMap{:name "application/json",
                                                                              :type "application",
                                                                              :subtype "json",
                                                                              :parameters {},
                                                                              :quality 1.0}}
                                   {:media-type #yada.media_type.MediaTypeMap{:name "application/edn",
                                                                              :type "application",
                                                                              :subtype "edn",
                                                                              :parameters {},
                                                                              :quality 1.0}}],
                        :methods {:post {:response #object[api.core$eval40522$fn__40524
                                                           0x41e5148f
                                                           "[email protected]"],
                                         :produces [{:media-type #yada.media_type.MediaTypeMap{:name "application/json",
                                                                                               :type "application",
                                                                                               :subtype "json",
                                                                                               :parameters {},
                                                                                               :quality 1.0}}],
                                         :parameters {:form {:email java.lang.String, :password java.lang.String}}}},
                        :show-stack-traces? false}

kwladyka23:01:15

I add there only :show-strack-traces? false

borkdude23:01:50

try moving the consumes into the post map, like in the test

kwladyka23:01:22

I was trying that one too

kwladyka23:01:33

I give up. I am going sleep.

kwladyka23:01:40

Thank you for help!