Fork me on GitHub
#clojure
<
2020-03-17
>
Prometheus11:03:33

I’m using @ptaoussanis sente for websockets both on the server and client. As I understand there is no way of setting the http headers. I have a separate step for login which uses regular http, and jwt tokens, using buddy. Sente does have the capability of having a user-id-fn which uses query-params. It is not secure to send tokens using query-params. I don’t want to send the user-id using this either. How should I go about this? securing my websocketconnection.

(sente/make-channel-socket-server! (get-sch-adapter)
                                           {:packer :edn
                                            :user-id-fn (fn [{:keys [params] :as req}]
                                                          (:userid (jwt/unsign (:token params)))))
This is what I currently have, but it’s not secure.

noisesmith17:03:14

> It is not secure to send tokens using query-params I thought that with SSL your params are hidden

noisesmith17:03:27

and if you aren't using SSL, nothing you are doing is secure

noisesmith17:03:20

I guess the params are safe, but using GET for sensitive content is still an issue https://stackoverflow.com/questions/323200/is-an-https-query-string-secure

Prometheus18:03:59

Ah ok so I’ll use the params, since I do use ssl of course

Setzer2211:03:08

Hi! Is there something I could use to create a sort of proxy repl? What I'd like is have something connect to a remote nrepl server and open a repl connection locally. The commands to this local nrepl connection would then be forwarded to the remote one.

vlaaad12:03:34

is nrepl a requirement?

vlaaad12:03:42

You can launch socket repls using system property, here are official docs: https://clojure.org/reference/repl_and_main#_launching_a_socket_server

vlaaad12:03:03

oh, sorry, you are talking about proxies…

Setzer2212:03:21

yes, socket repl wouldn't be an option unfortunately, since the point is to have cider work with this

noisesmith17:03:58

I just use ssh -L for this, and use a specific port, mapping it back to my host

noisesmith17:03:17

and cider can use socket repls (though not all features will be available of course)

noisesmith17:03:31

but yeah, ssh -L will let you use a remote nrepl

Setzer2211:03:49

I know they are less convoluted ways to do this, but it seems that unfortunately, I've hit some obscure bug in cider that won't allow me to connect to this specific remote host (which runs inside docker). The thing is a plain lein repl :connect works for me, as well as other editor plugins (e.g. VSCode's Calva). However, I'd like to keep using emacs for development, hence why I'm looking for this kind of workaround

pez12:03:24

What a strange bug. Asked about it in #cider ?

Setzer2212:03:05

yes, there's even an issue in the repo, but unfortunately it seems no one has an answer 😔

dominicm13:03:53

Netcat could do this

dominicm13:03:41

You would need one netcat which was listening in a port, piped into another netcat which connects to a remote host

kah0ona13:03:20

anybody knows how to configure logback.xml such that the correct line number shows up (using clojure.tools.logging)?, because if I use %L as a pattern it shows the line 286 of logger.clj, rather than the caller

ataggart18:03:33

I suspect the issue is that %L relies on stack inspection to determine the line number, and the clojure.tools.logging macros eventually bottom out at some reification of clojure.tools.logging.impl/LoggerFactory, which makes the implementation-specific method call.

kah0ona07:03:45

I feared this already. Oh well 🙂 I used Timbre before, which didn’t have this problem, for obvious reasons.

Brian13:03:37

Hi there. I am getting inconsistent CORS behavior when flipping between git hashes on a deps dependency that uses pedestal. I don't think this is a pedestal error however and I imagine there is some Clojure file that isn't being updated properly. Please read on. When I say "cors error" I mean that my typescript fetch() function is returning Access to fetch at '' from origin '' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource when hitting my Clojure API. In order to figure out where I broke things, I started checking out different commit hashes going backwards in time in order to find where things began working again. This led me to find very peculiar behavior. Please note that I am changing commit hashes on a library I am importing from my deps.edn file, not using git checkout <hash> . If the commit tree is like a>b>c with c being the most recent commit I'm able to repeatedly do this:   use commit hash a -> it works! use commit hash b -> it works! use commit hash c -> cors error! use commit hash b -> cors error! use commit hash a -> it works! use commit hash b -> it works! use commit hash c -> cors error! use commit hash b -> cors error! Notice how commit b works when it comes after commit a but does not work when it comes after commit a. I've also been able to go from commit a to commit c and get commit c to work. Perhaps some Clojure preferences are being stored somewhere on my machine and aren't properly overwritten when I change hashes? I'm not sure where to turn as I've seen both every hash fail but also every hash succeed.

dpsutton13:03:38

“I’ve seen every hash fail but also every branch succeed” what does this sentence mean?

Brian13:03:42

@dpsutton fixed. by "branch" I meant "hash"

orestis14:03:33

There’s very very rarely anything stored somewhere on your machine that would impact this. You are restarting the REPL when getting the new version, right?

orestis14:03:04

I’d cut out the typescript/fetch and use curl — curl -v should show you exactly what the response is, including the headers.

Brian14:03:13

Restarting my REPL every time yes. I am using chrome's network tab to see everything and there isn't anything enlightening there. And the cors error is such that I never see any response from the server. using curl I just get "connection refused" errors

orestis14:03:17

Well there you go — perhaps the server isn’t even starting?

orestis14:03:55

The browsers (Chrome etc) will first issue an OPTIONS request to the server, (pre-flight), and then only if the various CORS stuff is setup correctly, will they proceed to the actual request.

orestis14:03:15

curl bypasses all that and usually allows you to go to the actual cause of the issue.

Brian14:03:37

No it is starting. I do the same thing every time: stop repl, add new hash, start repl, start server, refresh chrome. I also can verify that the server has been started because I am able to query my API through postman in these cases

Brian14:03:29

My curl output in these cases (chrome can't preflight, server is up and working through postman) is

curl -v                                                                                   130 ↵
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Tue, 17 Mar 2020 14:58:04 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
Not Found%  

Brian15:03:44

Like I said though, as I change my git hashes it will start working. Sometimes git hash b will work, sometimes it won't. Same goes for commits a and commits c.

dpsutton15:03:39

you said that curl would respond with a connection refused?

Brian15:03:08

Yes see line 4 of the curl output

dpsutton15:03:47

I don’t believe that’s a cors issue then

orestis15:03:19

Oh that’s curl trying to connect over IPv6, it successfully tries localhost later.

orestis15:03:17

I think we could help much more if we show the code in question and also the chrome network tabs.

orestis15:03:40

curl indicates that there’s no GET handler under /api

Brian16:03:32

Yo @dpsutton @orestis so I found a bug in my code which caused the API to error out which looked like a CORS error. My best guess at the hash changes is that I think my REPL somehow retained state in some way. I'd stopped my repl, rebooted it with a new hash before that bug was introduced, and still saw that same bug. I rebooted IntelliJ and started a new REPL, and this time the bug was gone even though the git hash hadn't changed. This isn't the first time Cursive/IntelliJ has really screwed me up

😟 4
dpsutton17:03:54

I'm reading the doc string for clojure.edn/read-string and it just kinda defers to the edn spec. There is no mention of metadata in the edn spec as far as i can tell but metadata seems to be handled. (edn/read-string "(ns ^{:foo {:bar :baz}} bug.foo)") returns (ns bug.foo)

dpsutton17:03:12

is this defined behavior?

dominicm17:03:24

The clojure edn reader doesn't fail on undefined behavior, be careful

dominicm17:03:37

There's other things where it produces undefined behavior

noisesmith17:03:47

@dpsutton try calling pr-str with *print-meta* set to true

dpsutton17:03:32

yeah i see it has the meta. My questions is does edn support the presence of meta. Meta appears nowhere in https://github.com/edn-format/edn

noisesmith17:03:41

user=> (binding [*print-meta* true] (pr-str (edn/read-string "(ns ^{:foo {:bar :baz}} bug.foo)")))
"(ns ^{:foo {:bar :baz}} bug.foo)"

noisesmith17:03:01

yeah, I don't know if it's an intentional feature

dpsutton17:03:16

ha same 🙂

dpsutton17:03:56

bug in CIDER when there's nested maps in the meta on an ns form. we were using a regex to parse the ns form but that's not gonna fly if there can be nested maps

dpsutton17:03:23

an edn parser in elisp doesn't handle the meta. but wanted to find a spec that said it should

dominicm17:03:58

If the spec doesn't say so, then it doesn't.

orestis17:03:45

I was trying to figure out how to use EDN to round-trip some data that includes a few custom classes, but it seems that it’s fraught with pitfalls and lack of clarity on how to do it (see https://www.nitor.com/en/news-and-blogs/pitfalls-and-bumps-clojures-extensible-data-notation-edn) — I ended up using Transit which seems to be much more well suited…

💯 8
noisesmith17:03:17

yeah, transit is what I ended up with too, it works great

noisesmith17:03:12

I made this lib for quickly turning repl experimants into unit / regression tests: https://github.com/noisesmith/poirot

dominicm18:03:42

Transit is not suitable for storage :) > NOTE: Transit is intended primarily as a wire protocol for transferring data between applications. If storing Transit data durably, readers and writers are expected to use the same version of Transit and you are responsible for migrating/transforming/re-storing that data when and if the transit format changes.

👍 4
dominicm18:03:58

Transit JSON far outperforms edn afaik, I'll be benchmarking that soon I think.

mike_ananev18:03:31

The half of the article is not true

mike_ananev18:03:50

Use EDN or Transit and there will be no problems.

noisesmith18:03:41

transit hasn't actually broken compatibility yet, and it's from a team that very rarely does so

lilactown18:03:54

the transit lib itself has changed, though, so upgrading it in the event of a breaking change can be difficult

Alex Miller (Clojure team)19:03:46

This is silly given that this hasn’t happened in the 7 years or whatever since it was released

orestis19:03:22

I would like a way to write EDN without implementing protocols for things I don’t own (java.time, Mongo ids).

noisesmith19:03:23

yeah, I think transit is the best existing solution - though doesn't edn itself allow extension via multimethod? I prefer transit as it's a local data config rather than a global registry though

Alex Miller (Clojure team)19:03:54

edn uses the printer, which is extensible via multimethods for types you don’t own

noisesmith19:03:22

and the reader is configured inline, you just give it a map from tag to impl

orestis06:03:25

Yeah the reader is configured inline - what I want is to configure the writer inline, because a multimethod impl will affect my whole app, and it feels like a heavy handed approach. Perhaps I miss some guidance on the subject though. Is it always a good thing to support printing for common types used within the codebase?

noisesmith14:03:34

yeah, you get inline writer config out of the box with transit, I don't know if there's a good way to get local override of multimethod dispatched printing - use a thread local and a function that uses the normal behavior if it's not set?

noisesmith14:03:42

that feels hacky

Alex Miller (Clojure team)14:03:02

one of the items that we're considering for clojure 1.11 is the general issue of edn print/read roundtrip and this seems like a useful problem case. could you share what types you'd want to customize and why?

orestis15:03:21

Sure! Here’s what my current transit write-handlers look like:

{org.bson.types.ObjectId (transit/write-handler "mongo-oid" (fn [x] (str x)))
   com.mongodb.DBRef (transit/write-handler "mongo-ref" (fn [^com.mongodb.DBRef x] {:collection (.getCollectionName x)
                                                                                    :_id (.getId x)}))
   java.time.Instant (transit/write-handler "java.time.Instant" (fn [x] (str x)))}

orestis15:03:20

java.time.Instant is well-known I guess — the other two is some id-like stuff from mongo, which upon reading should be re-hydrated to the same types.

Alex Miller (Clojure team)15:03:55

seems like the mongo cases would likely be fine to print to custom tagged literals (and read via the same)?

orestis15:03:02

Going even a step back, I (right now) want a transient serialization format that is easy to produce and consume from Clojure. I do some local prepping and munging of data, then I upload to a remote server for them to be inserted into the database. If I can’t maintain the types across the wire it’s a lot of hairy code to write.

orestis15:03:41

In day-to-day handling of these values, it’d be nice to copy-paste print output from say CIDER into my code when playing around.

Alex Miller (Clojure team)15:03:57

well, transit was designed for this use case

Alex Miller (Clojure team)15:03:15

sorry, message interleaving, but the wire case

orestis15:03:18

Yes, custom tagged literals should work just fine. But I’m hesitant to implement it right now since it will be across my entire codebase…

orestis15:03:12

I’m not well versed in the arts of the Lisp, so perhaps I should for every common-used type in my codebase define a print-method (tagged literals) and data_readers?

Alex Miller (Clojure team)15:03:47

it depends on what you're going to do with that data

orestis15:03:48

So there’s the day to day dev, where printing stuff like

:tags
 [#object[com.mongodb.DBRef 0x6fd5ffa3 "{ \"$ref\" : \"tags\", \"$id\" : \"5e70e2f2680752cf5f88149e\" }"]],
 :status_id
 #object[org.bson.types.ObjectId 0x3ca08359 "538dc3e80f942a4767ffcd76"],
is an eye-sore, and if you want to take a value from that for further exploration, I can’t just read back.

orestis15:03:45

The thing that bothers me right now with EDN and tagged literals (but I don’t have anything concrete other than a hunch) is the fact that multimethods are global and non-overridable, and data_readers feels … weird. (although I can use clojure.edn/read which can have inline data readers AFAICT). Not sure if my hunch is correct or not.

Alex Miller (Clojure team)15:03:56

you can bind data readers in dynamic scope too if using the normal data reader (and have fallback to default-data-reader-fn, etc) so I think you have a lot of dynamic control on the reading side

Alex Miller (Clojure team)15:03:45

so none of the reading side stuff seems weird to me. I do think having something more dynamic than a multimethod would be interesting on the print side

Alex Miller (Clojure team)15:03:21

you probably could make that yourself with a custom printer for Object that deferred to a dynamic var

Alex Miller (Clojure team)15:03:51

that would kind of be the inverse to readers

orestis16:03:18

Apart from the theoretical neatness of being able to round-trip EDN, what other benefits are there? Honest question, looking to learn something new.

noisesmith17:03:03

from my perspective I think that's more than just neatness - the utility of EDN is the round trip capability and having better symmetric control of how the round trip happens is the difference between relying on clojure.edn vs. using another lib like transit

noisesmith17:03:39

I have no complaints about transit, but having something with the features I want in clojure itself is great

Alex Miller (Clojure team)18:03:06

if you are looking to round-trip edn (for example in human readable files, not on wires to programs) then it can be more challenging than it should be right now to ensure that the edn you write (via pr methods) is readable. there are a lot of ways to attack that problem (validate as data, validate while printing, or even prevent creation in the first place). we have not yet done any work on it.

mac23:03:41

I am using clojure-mail to download mail from my server. Mails with attachments provide a com.sun.mail.util.BASE64DecoderStream. This is an instance of a http://java.io.InputStream. How would I write this safely to disk?