Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-04-08

Channels

#ring
<
2022-04-08
>
stephenmhopper22:04:18

Hi, I have a ring + reitit app. One of the endpoints downloads a file from S3 and sends it to the user. I’m using ring-io/piped-input-stream . In my function, I call S3 to get an InputStream on the resource. And then I do something like this: 

(with-open [input-stream (:input-stream response)]
  (io/copy input-stream output-stream))
1. Is there a better way to implement all of this? 2. It seems like the entire input-stream is being downloaded to the server before anything is sent to the client. Is there a way around that?

stephenmhopper22:04:51

> If you aren’t doing any processing on the file, can the user just retrieve it directly from S3 via URL, without you having to shuttle the data through your app? @UTFAPNRPT I explored that path, but I can’t make the files in the bucket publicly available so that unfortunately won’t work.

wevrem00:04:01

I'm going to be needing something similar, and I was going to use the “authenticated” URL’s from S3 (not sure if that is the correct term) that give access to non-public buckets for a short duration. Haven't implemented it yet.

stephenmhopper01:04:31

Oh, that would fit my use case. I’ll have to look into that.

stephenmhopper01:04:35

As far is Ring is concerned though. Suppose I have an arbitrary InputStream, what’s the best way to pipe that to the client?

hiredman04:04:09

to some degree this is a limitation of http, you must set a content-length and a stream doesn't come with a content-length

hiredman04:04:30

it is also a limitation of some aws libraries, if I recall cognitect's aws-api for example will download the entire content in memory and wrap it in an inputstream

hiredman04:04:01

so the data being resident in memory may not be a result of ring stuff at all

stephenmhopper12:04:15

That’s good to know. Previously, I was working on a project where I needed to stream files stored in Postgres down to a client. The backend was written in Scala+Play+Slick. So I https://github.com/tminglei/slick-pg/pull/295 to allow for reactive streaming of data from the Postgres LargeObject API to anything that wanted to consume that Source. (The DB isn’t super important here, I’m more interested in creating reactive, consumable streams of bytes) That was six years ago. Is there not some equivalent to this in Clojure? If not, I might consider rolling something that other folks can re-use in the future. I just want to avoid re-inventing the wheel. 1. Does Ring support chunked requests? I couldn’t find official documentation on this, but it https://github.com/ring-clojure/ring/blob/4292f995a25895d0822f1fd040a1fbf89a17eeb1/ring-jetty-adapter/test/ring/adapter/test/jetty.clj. 2. For the whole “copying from InputStream to OutputStream reactively” part, should I just set up a core.async channel between the two streams that copies bytes in batches? That sounds slow to me. Is there a better way to go?

wevrem19:04:20

I think an InputStream is acceptable as the :body of a response.

wevrem22:04:21

If you aren’t doing any processing on the file, can the user just retrieve it directly from S3 via URL, without you having to shuttle the data through your app?