Fork me on GitHub
#clojure-uk
<
2018-01-05
>
yogidevbear09:01:31

Morning gentle people

benedek09:01:07

@seancorfield can you point me to the issue about trans deps by zach oakes pls?

paulspencerwilliams12:01:40

Anyone familiar with buddy, JWTs, and Ring? I'm knocking together a PoC, and I've been referencing https://github.com/funcool/buddy-auth/blob/master/examples/jwe/src/authexample/web.clj which works okay, but when I try to split the route into a seperate ns, what would be a good way of sharing the nonce between NSs as it's needed when invoking jwt/encrypt and jwe-backend which I'd normally put into handler.clj like

(def auth-backend (jwe-backend {:secret secret
                                :options {:alg :a256kw :enc :a128gcm}}))
(def app
  (-> (routes home-routes health-check-routes auth-routes app-routes)
      (handler/site)
      (wrap-authorization auth-backend)
      (wrap-authentication auth-backend)
      (wrap-json-response {:pretty false})
      (wrap-base-url)
      (wrap-cors :access-control-allow-origin [#""]
                 :access-control-allow-methods [:get :put :post :delete])))
I'm currently simply referencing the secret from the auth ns like
(:require [ourbucketlist-server.routes.auth :refer [auth-routes secret]])

maleghast15:01:33

@paulspencerwilliams - I am afraid I can’t help at all, as I never got my head around Buddy… I’ve been using JWT (I think) with Yada, and it’s entirely possibly that it’s using Buddy “under the hood” away from my need to understand it, but I don’t know one way or the other, I am afraid…

dominicm15:01:22

It does use buddy under the hood 😄

maleghast15:01:44

Of course it does… Typical… 😉

maleghast15:01:43

@paulspencerwilliams - What I was going to ask is why would you split your routes over more than one namespace..? This is not a snide, critical, “being a git” question, I really am curious.

paulspencerwilliams15:01:21

Good question @maleghast, suppose it's a hangover from Rails etc, where I'll have a set of routes for auth, one for products, one for admin etc.

maleghast16:01:33

@paulspencerwilliams - that explains that; I can totally grok that 🙂 Just a thought in terms of your issue, why not have all your routes and resources in one namespace and refer in functions from other namespaces to handle the “work” for the routes / resources so that your logic is organised the way that you want. That way you don’t need to share the secret(s) around, as the routes / resources would figure out authorisation in the central namespace?

maleghast16:01:45

(I am coming at this from a Bidi perspective where the routes can have clear separations into groups, but because all they are are connectors (in my case to Yada resources) ymmv if you are using another routing approach)

paulspencerwilliams16:01:16

@maleghast that would work. However, I've been using lein new compojure-app which creates a handler.clj and routes.clj, so although there's only 1 'routes' file, it's still seperated from building the 'app'. Obviously, I don't have to be consistent with what that template gives me, but maybe there's good reason for the template author's decision?

paulspencerwilliams16:01:58

I'd like to play more with Bidi, I hear good things but lack of time / life!

guy16:01:31

I’ve used bidi a little and found it really nice

guy16:01:45

mainly around the routing side of things

guy16:01:55

being able to do the bi-directional routing has been useful

guy16:01:01

i think thats what they call it

maleghast16:01:11

@guy - Yeah the name Bidi is a reference to the library being able to use the routes in both directions.

maleghast16:01:47

@paulspencerwilliams - There may indeed be a specific reason, I have no idea 🙂 Also it’s so long since I used Compojure I can’t remember whether or not you do any “work” inside the route definitions or not..?

maleghast16:01:44

Even if you do, you could always create a new namespace / file for your each of products, auth, admin and so forth and then just include them in the (ns …) at the top of routes.clj so that your routes are just referencing code from other namespaces.

paulspencerwilliams16:01:19

@maleghast actually, it was this lein template that suggested this approach. The template generated app looks like

(def app
  (-> (routes home-routes app-routes)
      (handler/site)
      (wrap-base-url)))

maleghast16:01:22

That would allow you to organise the logic for different bits of the app into different files, and you could pass secrets as parameters without having to make them available to the other namespaces

paulspencerwilliams16:01:19

Thus multiple routes, but actually playing, it would appear that multiple routes aren't additive or rather mutable. Later routes overwrite earlier ones in the list 😕

paulspencerwilliams16:01:25

Actually, earlier one wins.

maleghast16:01:08

That’s going to be annoying for you

paulspencerwilliams16:01:27

The previously available health-check routes become unavailable after preceeding with a non clashing route in bucket-list in

(def app
  (-> (routes home-routes bucket-list-routes health-check-routes auth-routes app-routes)
      (handler/site)
      (wrap-json-response {:pretty false})
      (wrap-json-body {:keywords? true :bigdecimals? true})
      (wrap-base-url)
      (wrap-cors :access-control-allow-origin [#""]
                 :access-control-allow-methods [:get :put :post :delete])
      (wrap-authorization auth-backend)
      (wrap-authentication auth-backend)))

paulspencerwilliams16:01:29

Swapping the order to health-check is first, shows those routes, and bucket-list routes become unavailable. Not sure why initial template works with multiple routes, I'll have to dig deeper. Not to fear, I'll try your suggestion of a larger route table, and split the functions by domain.

maleghast16:01:51

Interesting that it behaves that way - hope you have some luck with my suggestion at least! 🙂

maleghast17:01:55

Just in case there are different people online today… Does anyone have any recommendations (or “steer well clear of”) for tools that are Clojure-friendly (or not at all but highly useful) for planning / designing document-based data domains..?

maleghast17:01:12

Or should I just start hand-balling JSON schema?

maleghast17:01:49

(I’ve been doing a lot of reading yesterday and today but am still no closer to my own conclusion, so I am looking for some opinion(s) to tip the balance)

otfrom17:01:08

@maleghast I've always "just done it" but then I like text as a design tool for a lot of things (tho vis is nice)

maleghast17:01:51

@otfrom - I kinda like text as design tool, but I am wary of something that will be hard to manage as it grows in size…

maleghast17:01:01

…but I may have no other options.

maleghast17:01:35

And at least a large collection of well organised Schema can be in version control etc.

maleghast17:01:13

Also, I am worried that JSONSchema is too much of a moving target still… The more modern validator in Clojure-Land is only capable up to draft04 and the standard is on draft07 already, and it’s been around for ages and is still going through drafts…

guy17:01:23

I mean you could try spec?

maleghast17:01:38

But I want to use JSON as it will let me use JSONB in PostgreSQL and / or DynamoDB and several options in the noSQL canon, and it will allow me to share data with non-Clojure partners.

maleghast17:01:16

@guy - I could, but then I would be relying on Clojure to describe the data domain, and I expect to have to share data with non-Clojure environments in the near to medium terms

guy17:01:42

clojure is quite good at converting from edn to whatever you need as well

guy17:01:56

But i get what you are saying

guy17:01:59

:thumbsup:

maleghast17:01:26

That’s true @guy - I know what you mean…

maleghast17:01:03

Maybe I start with Spec and create JSONSchemas later…

guy17:01:31

I would say it depends on how much of your system is clojure

maleghast17:01:32

My worry there is that I will use spec too fully to be able to use JSONSchema, i.e. I will use something that JSONSchema cannot codify

maleghast17:01:02

@guy - All of it, but there is a good chance that people I will get data from and that I will need to send data to will not be.

guy17:01:38

I thought that is what spec was for though? That it would work at the boundries of your application

guy17:01:56

as it comes in, you can use spec, Then as you produce/send the data, you can use spec again to validate

guy17:01:02

and then convert i guess?

maleghast17:01:45

I suppose so, I struggle to see it as a way to define the data domain for an app / system though. I see it as a way of ensuring that nothing that is “wrong” gets shoved into a function.

maleghast17:01:58

But I would be happy to be shown the error of my ways…

guy17:01:35

From what i have understood, i would use spec to spec out your data, as it would look going through your system

guy17:01:56

So you could have a spec for the initial input, then when its converted and to each step

guy17:01:21

That way if you just do step by step transformations you can see what the data would look like at each step

guy17:01:47

I guess it depends on what you are doing

guy17:01:04

but for data transformation pipelines i would use it

guy17:01:37

I’ve not really explained your question so i apologise for that haha

maleghast17:01:09

The “thing” I am doing is trying to figure out how to design / map out the foundational data domain for our system, so that we can store and retrieve data for very specific things that will not change from client to client/

guy17:01:29

Yeah and a json schema and spec would look quite similar i would imagine

guy17:01:49

arnt they just hash-maps to predicates?

guy17:01:59

(my assumption)

maleghast17:01:00

so, if hypothetically you needed to store lots of broadly homogenous data about crops, example, and the broad plan was to do that in a JSON-based document store of some kind

maleghast17:01:11

then the raw tool choice (i.e. EDN / JSON / clojure-spec / Something Else(tm)) is important(ish) and along with it I am looking for any tools that make the process faster / simpler than hand-typing lots of Schema

maleghast17:01:36

I fear I may be looking for a tool that I am going to end up building and that other people will benefit from…

madstap17:01:30

Seems like they made that tool as well...

maleghast17:01:41

@madstap - Thanks, will go and have a read…

maleghast17:01:38

That is a really good / interesting article.

maleghast18:01:53

I may be veering towards clojure.spec, EDN and Datomic - this is not necessarily a bad thing at all… 😉

dominicm18:01:08

@guy @maleghast validate your boundary data before handing it to spec.

maleghast18:01:55

@dominicm - Can spec not do that validation?

maleghast18:01:06

(I really don’t know either way)

dominicm18:01:01

It's unsafe to.

maleghast18:01:42

Any recommendations on how best to safely validate the boundary data?

dominicm18:01:23

Let's say that data.xml registers a spec, and this spec does something O(n) which is about 10ms/character. But that's fine, this spec is generally used on strings of 1-5 chars. If I send edn to your endpoint which is being validated by spec with a :data.xml/foo key, where the string is 1000s of characters long, I will DOS your server will many of these kinds of requests.

otfrom08:01:49

I'm not sure I agree with this. Tho I do think you might want a different spec at the front door then you might in other parts of the system (as your msgs coming in might not be the same as your storage msgs or other things inside your system)

otfrom08:01:27

I'd be careful about what specs I was using to validate things coming in though. The O(n) 10ms hack is something that can hit any kind of incoming msg validator.

otfrom08:01:33

my first check on that would probably be something on max msg size for a string (that would make my API return a 400 or similar). After that I'd use spec and select-keys to get what I was interested in.

dominicm09:01:11

select-keys is a solution. It's also antithetical to spec's philosophy.

otfrom11:01:29

I'm not saying that select-keys would be part of the spec, but part of the input handling. Spec to check or other to check the length of the payload, then spec to check the format and then select-keys (if I wanted) to send things on in a Postel's law kind of way

dominicm13:01:12

You should select-keys first. Maybe after checking the input length, (although this is hard with certain types of data).

guy19:01:33

Interesting what would you use instead @dominicm?

dominicm19:01:52

@guy I don't have a full answer for you. I suspect the answer lies in the form libraries, as your API should probably have good messages. Depending on the intention. The fact is that spec doesn't allow you to restrict the keys in a map, so is unsuited to use on something a malicious user could access.

dominicm19:01:47

It's close to allowing a user to access your vars, but not quite.

dominicm19:01:58

@madstap @maleghast I forgot to tag you above

maleghast19:01:09

Thanks @dominicm - I got it when you did tag 🙂

maleghast19:01:07

And that’s quite surprising, but it makes sense now that you explain it.

otfrom11:01:29

I'm not saying that select-keys would be part of the spec, but part of the input handling. Spec to check or other to check the length of the payload, then spec to check the format and then select-keys (if I wanted) to send things on in a Postel's law kind of way