This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-03-09
Channels
- # aws (51)
- # beginners (57)
- # calva (10)
- # chlorine-clover (7)
- # cider (20)
- # clj-kondo (55)
- # clojure (43)
- # clojure-europe (9)
- # clojure-italy (1)
- # clojure-nl (5)
- # clojure-spec (8)
- # clojure-uk (71)
- # clojurescript (33)
- # core-async (22)
- # cursive (20)
- # datomic (3)
- # emacs (8)
- # figwheel-main (8)
- # fulcro (13)
- # garden (2)
- # graalvm (60)
- # graphql (26)
- # jobs (6)
- # joker (6)
- # kaocha (2)
- # lambdaisland (5)
- # malli (36)
- # off-topic (9)
- # portkey (15)
- # re-frame (3)
- # reagent (25)
- # remote-jobs (4)
- # spacemacs (3)
- # sql (111)
- # tree-sitter (29)
- # uncomplicate (3)
- # xtdb (2)
is "Throttling" represented as an http status code, or only within [:ErrorResponse :Error :Code]) ?@ghadi You asked the above. Throttling is only represented via that path. Here's a simple example:
(def c (aws-api/client {:api :workspaces}))
(def results
(vec (pmap (fn [_]
(aws-api/invoke c {:op :DescribeWorkspaceDirectories}))
(range 20))))
(first (filter anom/anomaly? results))
=>
{:__type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}
(get-in (meta *1) [:http-response :status])
=> 400
{"x-amz-date" "20200309T215544Z",
"x-amz-target" "WorkspacesService.DescribeWorkspaceDirectories",
"content-type" "application/x-amz-json-1.1",
"host" "",
"authorization" "..."}
{"x-amzn-requestid" "59bbfe02-e3b0-49fa-a2cb-2027a236531a",
"connection" "close",
"content-length" "58",
"date" "Mon, 09 Mar 2020 21:55:44 GMT",
"content-type" "application/x-amz-json-1.1"}
We also have to use this for :retriable?
for the apis we use.
(def api->retriable-fn
"Map of AWS API to a fn that will be called to see if the request should be retried
based on the HTTP response. This is needed because various AWS APIs indicate
failures in non-standard ways."
{:workspaces #(= "ThrottlingException" (:__type %))
:pricing #(= "ThrottlingException" (:__type %))
:monitoring #(= "Throttling" (get-in % [:ErrorResponse :Error :Code]))})
I'm trying to see if there is something generic we can detect and return a :busy anomaly
The body is this: {:type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}
(slurp (:body (:http-response (meta (first (filter anom/anomaly? results))))))
=> "{\"__type\":\"ThrottlingException\",\"message\":\"Rate exceeded\"}"
Yep... It's frustrating. The above retry works but it results in incorrect anomalies getting returned. In the above example, it returns incorrect when it should be busy.
@dchelimsky do we read :errors from the descriptors?
Not yet. We looked at it as a means of normalizing error handling, but there is nothing normalizable (every service does its own thing).
Also, I ran into a few cases in which the descriptors themselves listed error shapes in operation maps, but those weren't present in the shapes map.
@ghadi @kenny am I missing some value we'd get from reading them anyway?
There is no spec for this @ghadi
No consistency across services.
The error shape name is in the operation, which is already part of the model we use.
I think that should be in (ops client)
(don't have a repl up for this at the moment)
the ThrottlingException @kenny encountered isn't even in the workspaces service descriptor :)
That's what I'm talking about 😕
Can't exactly knock AWS for this. Nobody said "hey, use these descriptors to build your own SDKs" 😉
Can't break promises you don't make!
This all came up recently, btw: https://github.com/cognitect-labs/aws-api/issues/123
Having the ability to manually transform response bodies before the :retriable?
call would help us in this situation. The transformed response would also need to be returned to the caller. I realize it's not a general solution, but it's sounding like there is not one.
On a totally unrelated note, @ghadi you may recall I brought up the issue of an AWS API returning an invalid next-token. After a month or so, I finally got it again! Unfortunately the code to log the request-id had a typo in the key to lookup so I didn't get a request-id from the failure 😞 The next-token looks totally standard for this API. The system retrying the same call ~30s later and it worked. Quite odd behavior. Wonder if AWS has some sort of race condition.
{:__type "InvalidParameterValuesException",
:message "Invalid NextToken provided.",
:cognitect.anomalies/category :cognitect.anomalies/incorrect,
:next-token "af286f1d-36ac-4fd2-bd5a-fa79fbb72f4b",
:page-count 4}