Fork me on GitHub
#aws
<
2020-03-09
>
kenny21:03:24

is "Throttling" represented as an http status code, or only within [:ErrorResponse :Error :Code]) ?@ghadi You asked the above. Throttling is only represented via that path. Here's a simple example:

(def c (aws-api/client {:api :workspaces}))
(def results
  (vec (pmap (fn [_]
               (aws-api/invoke c {:op :DescribeWorkspaceDirectories}))
             (range 20))))

(first (filter anom/anomaly? results))
=>
{:__type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}

(get-in (meta *1) [:http-response :status])
=> 400

ghadi21:03:51

do you have the complete response headers?

kenny21:03:28

{"x-amz-date" "20200309T215544Z",
 "x-amz-target" "WorkspacesService.DescribeWorkspaceDirectories",
 "content-type" "application/x-amz-json-1.1",
 "host" "",
 "authorization" "..."}

ghadi21:03:45

that's the response?

ghadi22:03:03

seems like the request

kenny22:03:09

Oops, yeah

kenny22:03:18

{"x-amzn-requestid" "59bbfe02-e3b0-49fa-a2cb-2027a236531a",
 "connection" "close",
 "content-length" "58",
 "date" "Mon, 09 Mar 2020 21:55:44 GMT",
 "content-type" "application/x-amz-json-1.1"}

kenny22:03:56

We also have to use this for :retriable? for the apis we use.

(def api->retriable-fn
  "Map of AWS API to a fn that will be called to see if the request should be retried
  based on the HTTP response. This is needed because various AWS APIs indicate
  failures in non-standard ways."
  {:workspaces #(= "ThrottlingException" (:__type %))
   :pricing    #(= "ThrottlingException" (:__type %))
   :monitoring #(= "Throttling" (get-in % [:ErrorResponse :Error :Code]))})

ghadi22:03:05

what else is in the json response?

ghadi22:03:31

I'm trying to see if there is something generic we can detect and return a :busy anomaly

kenny22:03:39

The body is this: {:type "ThrottlingException", :message "Rate exceeded", :cognitect.anomalies/category :cognitect.anomalies/incorrect}

kenny22:03:51

From my experience, I don't think there is 😞

ghadi22:03:52

what about the raw response JSON?

kenny22:03:04

Just slurping the :body?

ghadi22:03:14

yeah, should be available

kenny22:03:32

(slurp (:body (:http-response (meta (first (filter anom/anomaly? results))))))
=> "{\"__type\":\"ThrottlingException\",\"message\":\"Rate exceeded\"}"

ghadi22:03:45

Thanks AWS™

kenny22:03:42

Yep... It's frustrating. The above retry works but it results in incorrect anomalies getting returned. In the above example, it returns incorrect when it should be busy.

ghadi22:03:18

should really send back a status 429

kenny22:03:36

Agreed. This happens in several of their apis though.

ghadi22:03:17

@dchelimsky do we read :errors from the descriptors?

dchelimsky22:03:33

Not yet. We looked at it as a means of normalizing error handling, but there is nothing normalizable (every service does its own thing).

dchelimsky22:03:48

Also, I ran into a few cases in which the descriptors themselves listed error shapes in operation maps, but those weren't present in the shapes map.

dchelimsky22:03:37

@ghadi @kenny am I missing some value we'd get from reading them anyway?

ghadi22:03:15

just need to be able to dispatch on the error shape name

ghadi22:03:29

is that always in __type?

ghadi22:03:44

we can't generically understand the errors without a lookup table

dchelimsky22:03:46

That's only a thing in some json services.

✔️ 8
dchelimsky22:03:04

There is no spec for this @ghadi

dchelimsky22:03:13

No consistency across services.

dchelimsky22:03:38

The error shape name is in the operation, which is already part of the model we use.

ghadi22:03:02

error shape name is useful enough

dchelimsky22:03:15

I think that should be in (ops client)

dchelimsky22:03:29

(don't have a repl up for this at the moment)

ghadi22:03:06

the ThrottlingException @kenny encountered isn't even in the workspaces service descriptor :)

😞 4
dchelimsky22:03:17

That's what I'm talking about 😕

dchelimsky22:03:12

Can't exactly knock AWS for this. Nobody said "hey, use these descriptors to build your own SDKs" 😉

kenny22:03:43

They should return standard http status codes though

ghadi22:03:35

it doesn't say s* about how those errors are encoded on the wire

dchelimsky22:03:06

Can't break promises you don't make!

kenny22:03:42

Having the ability to manually transform response bodies before the :retriable? call would help us in this situation. The transformed response would also need to be returned to the caller. I realize it's not a general solution, but it's sounding like there is not one.

dchelimsky22:03:48

Please feel free to make suggestions there in comments.

4
kenny22:03:54

On a totally unrelated note, @ghadi you may recall I brought up the issue of an AWS API returning an invalid next-token. After a month or so, I finally got it again! Unfortunately the code to log the request-id had a typo in the key to lookup so I didn't get a request-id from the failure 😞 The next-token looks totally standard for this API. The system retrying the same call ~30s later and it worked. Quite odd behavior. Wonder if AWS has some sort of race condition.

{:__type "InvalidParameterValuesException",
 :message "Invalid NextToken provided.",
 :cognitect.anomalies/category :cognitect.anomalies/incorrect,
 :next-token "af286f1d-36ac-4fd2-bd5a-fa79fbb72f4b",
 :page-count 4}

kenny22:03:38

I am determined to get AWS to fix this issue. Hopefully next time this error occurs I'll have a request id for them to debug this!