This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-09-18
Channels
- # aleph (45)
- # aws (4)
- # beginners (56)
- # boot (2)
- # cider (45)
- # clara (2)
- # cljs-dev (9)
- # cljsrn (31)
- # clojure (71)
- # clojure-dusseldorf (8)
- # clojure-gamedev (1)
- # clojure-italy (22)
- # clojure-nl (1)
- # clojure-russia (46)
- # clojure-sg (1)
- # clojure-spec (5)
- # clojure-uk (40)
- # clojurescript (30)
- # community-development (3)
- # cursive (17)
- # data-science (1)
- # datomic (18)
- # emacs (3)
- # figwheel (1)
- # fulcro (19)
- # hoplon (12)
- # jobs (5)
- # leiningen (42)
- # off-topic (12)
- # om (2)
- # onyx (41)
- # re-frame (19)
- # ring-swagger (1)
- # rum (3)
- # shadow-cljs (4)
- # specter (7)
- # unrepl (2)
- # vim (25)
- # yada (24)
@aengelberg @dm3 as a general rule, if there's an error somewhere in a stream transform or operation, the error will be logged and the stream will be closed
I get that it usually cleans up resources properly, but it does not propagate the fact that something went wrong. A stream closing could either mean success or failure.
core.async chans have a similar problem.
error propagation is a bit tricky in general, imagine if src
is being split across two different sinks, a
and b
more to the point, if you're feeding into a network socket and it closes, is that correct behavior or an error?
in order to talk about something succeeding at the application level, you need to define your success criteria using logic layered atop the transport layer, not in the transport layer
in fairness, I absolutely could have put!
fail with an error rather than return false if, say, map
fails
but I suspect that people aren't checking for that, and in practice it would just cause the same error to be logged all the way up the topology
Great points
this isn't to say that you couldn't create a richer error model, I'm sure other stream libraries do
but being the lowest common denominator across all stream abstractions makes that difficult
My original complaint stemmed from the fact that I wanted to convert InputStream
into (seq-of bytes)
, and byte-streams
was (unnecessarily?) using a manifold stream in the conversion path, and that conversion silenced errors such as amazon.s3.SocketTimeoutException
that I wanted the consumer of the lazy seq to see.
It was logging the error
which is great
which I need to happen for my job to die and mark itself as failure
it depends on the implementation
in this case I was reading from S3, not writing to it
right, so what happens if the connection closes halfway through downloading the value
but I have a different, separate use case that also suffered from this problem in which I was uploading each chunk as a batch to S3 which could potentially fail
because if so, that strikes me as a little weird, you wouldn't expect the InputStream to be doing that for you
good question, I'd have to think back to the last time I saw this error happen in prod
and this comes back to my original point of "you get some data, it's up to you to decide if it's the correct, full data"
I know you don't control the S3 client api, so I'm not really pointing the finger at you here
but with anything over the network, lack of an explicit error is not a guarantee that everything's correct, which is kind of why I haven't worried about this part of Manifold very much
I could easily reproduce this by just manually opening an inputstream, waiting out the timeout, and seeing what happens
so basically my takeaway here is that I need to figure out what exact error I want to be able to see before I complain about not seeing it
possibly, or at least consider that failing to process everything isn't always signaled by an exception
so having exception propagation doesn't actually mean all issues are surfaced, and may lead to thinking that things are correct when they aren't
I'm fairly confident that my S3 read sockettimeout issue was somehow surfacing through one of the InputStream methods. But I'll need to confirm since you make an interesting point that it might just close it instead