pedestal 2021-09-16 | Slack Archive

Hi all, can someone guide me to an example for http://pedestal.io/reference/streaming?

(with-open [w (io/writer "out.json")]
  (json/write
    (repeat 1e8 {:hello "world"})
    w))

you should end up with a out.json with ~1.7Gb¹ Sometimes you need to read/write things that are larger than your memory (Xmx, in JVM case) For example, if you want to send (repeat 1e9 {:hello "world"}) as a JSON in a HTTP response. 1e9 with a simple and imprecise math should take ~10Gb of memory If you try :body (json/write-str (repeat 1e9 {:hello "world"})), it will probably run out-of-memory, because it will try to allocate a 10Gb string all at once You can switch to streamed responses To do that, you can use :body (fn [out] (json/write (repeat 1e9 {:hello "world"}) (io/writer out))). This is a "Streaming" approach. It should work in a simple 500Mb machine. Another option, if your 10Gb JSON file is already saved in your disk, you can do :body (io/file "out.json") or :body (io/input-steram "out.json") To go async, you will need to return something like (go (assoc ctx :response {:body my-chan})), then (dotimes [i 1e9] (async/>! my-chan {:hello "World"})) ¹ This value may be different in your machine due disk blocks and other resource allocation things

simongray13:09:35

What is the point of the dev-interceptors ? I recently needed to circumvent CORS during development and I thought maybe the dev-interceptors might be the way to do it, since it adds an interceptor called dev-allow-origin, but all that interceptor does is delete the origin header from the request…? This doesn’t actually allow cross-origin requests AFAIK. It turns out that the real solution to allowing cross-origin requests during development is to set the :io.pedestal.http/allowed-origins key to (constantly true) (…or I guess to manually add an interceptor that sets the Access-Control-Allow-Origin header to *).

lispers-anonymous17:09:13

dev-interceptors also includes a generic error handler that will return information about any unhanded exception in the response body. I find that to be useful during development. I can't speak to the dev-allow-origin interceptor.

Fredrik13:09:23

The way I read the code, an empty "origin" header is passed through the CORS checker

Fredrik13:09:49

Have you tested it out?

simongray13:09:38

There’s no need, since I fixed it? In order to get a non-empty response in a modern browser you need to set Access-Control-Allow-Originn. The dev-interceptors do not do this.

simongray13:09:20

I was just commenting on the fact that this piece of code seemingly hasn’t kept up with modern browser policy.

simongray14:09:07

I just think that the dev-allow-origin interceport should set this header and it doesn’t do that. That would be my expectation.

Fredrik14:09:13

I see your point now, the problem is not that the request doesn't complete server-side, it's that the browser requires this header on the response

simongray14:09:52

Yeah. If you don’t set Access-Control-Allow-Origin the JS or CLJS app making the request will simply get back an empty 200 response. If you CURL the same resource it works fine, because CURL isn’t a web browser and doesn’t care about CORS.

lispers-anonymous17:09:14

Sounds like a great issue to open in the pedestal repo.

2021-09-16

Channels