This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-10-06
Channels
- # aleph (15)
- # announcements (2)
- # babashka (121)
- # beginners (62)
- # biff (6)
- # cherry (2)
- # cider (51)
- # clerk (30)
- # cljs-dev (5)
- # clojure (77)
- # clojure-austin (2)
- # clojure-europe (10)
- # clojure-germany (6)
- # clojure-nl (1)
- # clojure-norway (19)
- # clojure-romania (1)
- # clojure-uk (3)
- # clojurescript (16)
- # core-typed (7)
- # cursive (17)
- # datomic (12)
- # deps-new (11)
- # emacs (7)
- # events (2)
- # fulcro (5)
- # honeysql (2)
- # hyperfiddle (32)
- # introduce-yourself (1)
- # jobs-discuss (2)
- # membrane (18)
- # missionary (2)
- # music (5)
- # polylith (7)
- # reagent (26)
- # releases (5)
- # testing (32)
- # tools-build (14)
- # tools-deps (7)
- # xtdb (8)
I'd like to get a sense of how people are using the pipeline-transform
option. If you're using it for anything, mind telling us a bit about what you use it for?
It's shaping up to be one of the truly non-backwards-compatible parts of the HTTP/2 code, since thanks to multiplexing, there will now be one shared connection-level pipeline, and one stream-level pipeline for each stream.
My standard usecase for it is adding HttpObjectAggregator to the pipeline, both for server and client connections.
Oh, interesting. Why do you add it? Is it to avoid dealing with InputStreams that are still being written to? Or something else?
When people set max-request-body-size
, Aleph adds it by default, if that's your use case.
Yes, it's both for being able to safely slurp the :body inputstream without worrying about blocking, and for limiting the max body size. I've put these middlewares to pipeline-transform back when the max-request-body-size
was not done/merged, and the code kinda stuck around.
To copy the "Expect" header before it got removed by the continue-handler https://github.com/clj-commons/aleph/pull/686#issuecomment-1673837252
(Edge case of the S3 API which supports only HTTP1.1 AFAIK)
@U06PNK4HG We could make aggregation a more general option, so people don't have to fiddle with pipeline-transform.
I thought most people were using bs/to-string
or similar fns to avoid dealing with InputStreams, but maybe that's not true?
bs/to-string
would allocate an extra string which I don't really need since I'm going to parse the JSON body into maps anyway.
The changes won't affect me directly since we stay on our fork of Aleph. I'm just voicing out the usecase, I certainly don't know if the way we do it right now is the best one:)
Well, one thing's for certain: for anyone using HTTP1 objects/handlers in their pipeline-transform
, which is probably everyone, none of that will work unaltered. The impls are a just a little too different, Netty doesn't share 99% of the code between h1 and h2, not even interfaces.
One solution for many people will be to insert the Http2StreamFrameToHttpObjectCodec
at the top of the stream channel. However, I don't think I can do that automatically in case users want to do something with the H2 objects.
Other issues are deciding between connection-level vs stream-level pipelines and sharing a stateful h1 handler across multiple stream-level pipelines.
I think it's fair that anyone mucking around with Netty pipelines shouldn't be isolated from the upcoming changes. To that end, I'm thinking of making three options:
1. http1-pipeline-transform - the current pipeline-transform will be made an alias for this
2. http2-conn-pipeline-transform - will alter the shared connection pipeline
3. http2-stream-pipeline-transform - will alter each h2 stream pipeline
I think as a safety check, if you enable h2 and set pipeline-transform
, we should throw an error if you haven't set either http2-conn-pipeline-transform
or http2-stream-pipeline-transform
.
I also think an aggregation flag might be handy, maybe aggregate-data
, wait-for-body-to-finish
or something, combined with some timeout and/or min amount of data before proceeding, lest it hang on a large/infinite body.
Thoughts?
Wrote up an issue at https://github.com/clj-commons/aleph/issues/692, pelase add comments there
re safety check: that error would not be thrown when only setting http1-pipeline-transform
instead? Would make sense IMHO if you intentionally want to modify the pipeline only for HTTP 1 for some reason.
as for pipeline-transform
uses: We use it for adding a read timeout handler. This turns out to be trickier than expected due to HTTP keep-alive. I'll elaborate on the ticket. (never mind, that's due to an implementation detail on our end)
Do the idle handlers not meet your needs for some reason?
Right, that's what I was hinting at: With connection keep-alive (especially so when sitting behind a proxy), setting the timeout low enough to serve as a read timeout, the connection would often get closed in between two requests for no good reason. In our particular case, we're doing long polling where we keep a requests open for up to 5 seconds. Thus, we have to set idle timeout to something higher than 5 seconds. But that is way too high to serve as a read timeout at the same time.
Well, one thing's for certain: for anyone using HTTP1 objects/handlers in their pipeline-transform
, which is probably everyone, none of that will work unaltered. The impls are a just a little too different, Netty doesn't share 99% of the code between h1 and h2, not even interfaces.
One solution for many people will be to insert the Http2StreamFrameToHttpObjectCodec
at the top of the stream channel. However, I don't think I can do that automatically in case users want to do something with the H2 objects.
Other issues are deciding between connection-level vs stream-level pipelines and sharing a stateful h1 handler across multiple stream-level pipelines.
I think it's fair that anyone mucking around with Netty pipelines shouldn't be isolated from the upcoming changes. To that end, I'm thinking of making three options:
1. http1-pipeline-transform - the current pipeline-transform will be made an alias for this
2. http2-conn-pipeline-transform - will alter the shared connection pipeline
3. http2-stream-pipeline-transform - will alter each h2 stream pipeline
I think as a safety check, if you enable h2 and set pipeline-transform
, we should throw an error if you haven't set either http2-conn-pipeline-transform
or http2-stream-pipeline-transform
.
I also think an aggregation flag might be handy, maybe aggregate-data
, wait-for-body-to-finish
or something, combined with some timeout and/or min amount of data before proceeding, lest it hang on a large/infinite body.
Thoughts?