Fork me on GitHub
#aws
<
2020-03-06
>
kenny18:03:37

Sometimes aws-api will return an anomaly with an incorrect category (e.g., incorrect or fault) when the actual category should be busy. This is on AWS for returning the wrong status code. We would, however, like to change the categories of these anomalies to the correct category. For example, the CloudWatch API will indicate a throttling error like so #(= "Throttling" (get-in % [:ErrorResponse :Error :Code])). Is there any thinking on the best way to do this? It would seem like the best way is to create our own invoke function and ensure every caller uses that instead of cognitect.aws.client.api/invoke.

Jimmy Miller18:03:41

When trying to use s3 :SelectObjectContent I get an error about failing to parse xml

Jimmy Miller18:03:41

(aws/invoke s3
            {:op :SelectObjectContent
             :request
             {:Bucket bucket
              :Key my-key
              :Expression "select * from S3Object s"
              :ExpressionType "SQL"
              :InputSerialization {:CSV {:RecordDelimiter "\r\n"}}
              :OutputSerialization {:JSON {:Type "LINES"}}}})

;; =>

{:cognitect.anomalies/category :cognitect.anomalies/fault,
 :cognitect.aws.client/throwable #error {
 :cause "ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog."
 :via
 [{:type javax.xml.stream.XMLStreamException
   :message "ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog."
   :at [.apache.xerces.internal.impl.XMLStreamReaderImpl next "XMLStreamReaderImpl.java" 652]}]
 :trace
 [[.apache.xerces.internal.impl.XMLStreamReaderImpl next "XMLStreamReaderImpl.java" 652]
  [clojure.data.xml.jvm.parse$pull_seq$fn__29940 invoke "parse.clj" 78]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.RT seq "RT.java" 535]
  [clojure.core$seq__5402 invokeStatic "core.clj" 137]
  [clojure.core$seq__5402 invoke "core.clj" 137]
  [clojure.data.xml.tree$seq_tree$fn__29773 invoke "tree.clj" 39]
  [clojure.lang.LazySeq sval "LazySeq.java" 42]
  [clojure.lang.LazySeq seq "LazySeq.java" 51]
  [clojure.lang.LazySeq first "LazySeq.java" 73]
  [clojure.lang.RT first "RT.java" 692]
  [clojure.core$first__5384 invokeStatic "core.clj" 55]
  [clojure.core$ffirst__5394 invokeStatic "core.clj" 103]
  [clojure.core$ffirst__5394 invoke "core.clj" 103]
  [clojure.data.xml.tree$event_tree invokeStatic "tree.clj" 70]
  [clojure.data.xml.tree$event_tree invoke "tree.clj" 66]
  [clojure.data.xml$parse invokeStatic "xml.clj" 109]
  [clojure.data.xml$parse doInvoke "xml.clj" 84]
  [clojure.lang.RestFn invoke "RestFn.java" 486]
  [cognitect.aws.util$xml_read invokeStatic "util.clj" 148]
  [cognitect.aws.util$xml_read invoke "util.clj" 145]
  [cognitect.aws.shape$xml_parse invokeStatic "shape.clj" 217]
  [cognitect.aws.shape$xml_parse invoke "shape.clj" 214]
  [cognitect.aws.protocols.rest$parse_body invokeStatic "rest.clj" 243]
  [cognitect.aws.protocols.rest$parse_body invoke "rest.clj" 235]
  [cognitect.aws.protocols.rest$parse_http_response invokeStatic "rest.clj" 260]
  [cognitect.aws.protocols.rest$parse_http_response invoke "rest.clj" 249]
  [cognitect.aws.protocols.rest_xml$eval32111$fn__32112 invoke "rest_xml.clj" 23]
  [clojure.lang.MultiFn invoke "MultiFn.java" 239]
  [cognitect.aws.client$handle_http_response invokeStatic "client.clj" 49]
  [cognitect.aws.client$handle_http_response invoke "client.clj" 44]
  [cognitect.aws.client$send_request$fn__31513$state_machine__20194__auto____31540$fn__31543 invoke "client.clj" 112]
  [cognitect.aws.client$send_request$fn__31513$state_machine__20194__auto____31540 invoke "client.clj" 108]
  [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic "ioc_macros.clj" 973]
  [clojure.core.async.impl.ioc_macros$run_state_machine invoke "ioc_macros.clj" 972]
  [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic "ioc_macros.clj" 977]
  [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invoke "ioc_macros.clj" 975]
  [clojure.core.async.impl.ioc_macros$take_BANG_$fn__20212 invoke "ioc_macros.clj" 986]
  [clojure.core.async.impl.channels.ManyToManyChannel$fn__15077$fn__15078 invoke "channels.clj" 95]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 628]
  [clojure.core.async.impl.concurrent$counted_thread_factory$reify__14946$fn__14947 invoke "concurrent.clj" 29]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 830]]}}

Jimmy Miller18:03:41

I'm guessing that because the aws-api is generic it assuming this endpoint returns xml when it does not. Is there anyway to make it not try to parse the body as xml?

ghadi18:03:36

@jimmy capture the (meta response)

ghadi18:03:44

should have a response body that might be helpful

ghadi18:03:12

(but obviously it's CSV)

ghadi18:03:03

can you file an issue? In the meantime, maybe try re-reading the response inputstream body correctly in userspace

ghadi18:03:12

as a workaround

Jimmy Miller18:03:40

Thanks for the work around. I will definitely file an issue about it

Jimmy Miller18:03:53

Any info you want me to capture other than what I've provided?

ghadi18:03:55

@kenny same request for you -- is "Throttling" represented as an http status code, or only within [:ErrorResponse :Error :Code]) ?

ghadi18:03:27

@jimmy any setup code necessary to make an object that is readable in this way

👍 4
dchelimsky18:03:40

Before you file an issue - please check your dependencies.

ghadi18:03:23

this one isn't probably our favorite data.xml dep issue --

ghadi18:03:36

but knowing the deps will be useful

ghadi18:03:53

no, he's asking for a special response handling from S3:

:InputSerialization {:CSV {:RecordDelimiter "\r\n"}}
              :OutputSerialization {:JSON {:Type "LINES"}}}})

ghadi18:03:08

dunno if that conflicts with the client's reading of the service descriptor

ghadi18:03:19

looks like this is a Two-fer Tuesday

rich 4
🙃 4
dchelimsky19:03:18

Type is not part of the JSON map under OutputSerialization

Jimmy Miller19:03:25

True, but changing that does not make a difference.

dchelimsky19:03:04

Are you using lein or tools.deps or other?

dchelimsky19:03:57

what version clojure/data.xml is in clj -Spath

dchelimsky19:03:07

OK - never mind. I see I'm barking up the wrong tree.

Jimmy Miller19:03:13

It is 0.2.0-alpha6. But yeah, if you look at the response body it isn't just straight forward xml. I'll file an issue that given a bucket you should be able to test it and recreate it. Sadly the body is complicated enough, probably not worth me trying to do the work around right now.

ghadi19:03:47

the response format is..... artisanal

😂 4
dchelimsky19:03:46

Or custom, in which case you could customize it. But how do you artisanize it?

ghadi19:03:51

:eventstream true