Fork me on GitHub
#aleph
<
2017-09-15
>
aengelberg17:09:48

The lack of error propagation in manifold streams makes me less confident in writing servers that use stream conversion to handle incoming data. For example, I would like to write a handler that takes an InputStream, converts it into a stream of chunks (easy enough with byte-streams), then upload each of those chunks to S3 in a multipart upload. But what if the S3 API throws an error when uploading one of the chunks? The stream consumer will silently fail, but won't stop the upload. Any way to reconcile this lack of error propagation?

dm317:09:28

Yeah, I’ve registered an issue https://github.com/ztellman/manifold/issues/95 a while ago

dm317:09:54

it even references a commit with an error-propagation implementation

dm317:09:21

@ztellman said that his experience with Lamina proved that attaching error handling to streams is more trouble than it’s worth.

dm317:09:38

I disagree - for the same reasons you have, @aengelberg

aengelberg17:09:37

I'm not convinced it's an easy problem to solve, either 🙂 In general, you'd ideally like to be able to both propagate errors forward and backward. e.g. If something hands you a stream, you'd like to know when the upstream processor stops due to failure. And if you hand something a stream, you'd like to know when the downstream processor fails.

dm318:09:07

well, I’ve implemented propagation downstream

dm318:09:27

upstream is a harder problem as sources don’t usually care about consumers

dm318:09:44

I don’t think you can really get by without the feature - in many parts of my code I’ve resorted to passing tuples of (Stream, Deferred[Error]) through combinators

dm318:09:55

Rx* libs have this too

shader21:09:09

another stream library I like a lot, most.js, provides a recoverWith method: https://github.com/cujojs/most/blob/master/docs/api.md#recoverwith

shader21:09:43

it's an interesting solution, in that the error handler function is expected to provide a replacement stream

shader21:09:40

in their example, you fall back to default data if you can't fetch data from a server