aleph

Matthew Davidson 2023-05-05T07:23:20.794639Z

Another quick question: Are you using multipart bodies on either the client or server side (e.g., using :parts to upload multiple files at once)? > We ARE using multipart > We ARE NOT using multipart React with the relevant emoji

✅ 2
❌ 3
Matthew Davidson 2023-05-05T07:26:26.749499Z

(If you're unsure, the answer is probably no)

Matthew Davidson 2023-05-08T10:01:27.506839Z

Hmmm. Unfortunately, Netty (which Aleph is built on), doesn't have any support for multipart in their HTTP2 code. It's not forbidden in HTTP2, but it also makes less sense, since you can just open up as many HTTP2 streams as you need to upload each file separately. We've directly supported multipart, but if we need to persist to disk, it was always a little awkward trying to craft a default retention policy and get the user to remember to handle it. Given all of this, middleware might be preferable, even if not Ring's. The alternatives are: 1. Throw an exception if using HTTP2 and multipart (nobody wants this, and it's not feasible on server side because you've already negotiated the protocol before you've seen the body) 2. Automatically downgrade to HTTP1 on client when multipart detected in Ring request (probably surprising and unwanted) 3. Insert the conversion codec to use HTTP1 objects in the Netty pipeline so existing multipart code works (slower, more garbage generated) 4. Same as above, but only insert the codec dynamically when multipart is detected (won't pay the price all the time, but requires detection code, buffering buffers, and the ability to lie about/switch the protocol used to manipulate which fns are being called) 5. Use Ring multipart middleware (would have to see how the Ring middleware will interact) 6. Import some non-Netty multipart lib for our own middleware/code, since netty really doesn't make it easy to use independently of the http1 code (another dep) 7. ...?

valerauko 2023-05-08T13:04:12.740219Z

oh is this limited to http2?

valerauko 2023-05-07T11:00:51.826879Z

heavy use of multipart upload handling on the server side (image processing, csv batch processing). using the ring middleware though, not the multipart features of aleph

valerauko 2023-05-07T11:04:37.173179Z

it'd be great if we could handle with "just" aleph since having to pull in the entire ring-core package with all its massive dependencies is making me a sad panda 🐼

Matthew Davidson 2023-05-19T13:40:33.232319Z

Essentially went with #3 above. We fall back to using the old HTTP1 code for multipart

Matthew Davidson 2023-05-09T07:49:19.413179Z

Well, my question was about multipart in general, but we're trying to decide what to do with it for the new HTTP/2 code.

valerauko 2023-05-10T01:45:28.882509Z

from the server developer's perspective how would it look like if aleph received a http/2 request equivalent to a multipart = multiple streams?

Matthew Davidson 2023-05-10T05:36:59.052189Z

You mean, could we create multiple streams on the server, one per part, and call the user's handler for each one? Hmmm. That will be more work than supporting multipart in a single handler, I think. If the files are always processed independently, it wouldn't be so bad. But if you're relying on them all existing at once, or need to know some are ready before processing others, concurrent handlers will make coordination harder. Plus, we'd need to pick a response format (single response, streaming array of individual responses, etc) and an error-handling policy, and whatever we choose might not work for all situations. Some of the headers would be incorrect and require adjustment for each part (content types, lengths, etc). We could give a new "multipart-handler" just the files, no headers, but sometimes people will still need headers. At best, I think we'd support the same interface as the current decode-reqeust; a stream of the files in order as they're decoded. If users can process them in parallel, they can do that manually.