aws

cch1 2025-05-20T00:01:57.131839Z

I've been so happy with the aws api's stability. Today, for the first time in years, I did see an error that I'm pretty sure is due to a bug in the aws-api or perhaps a "spec failure" from AWS itself. Details in the 🧵 .

cch1 2025-06-21T00:52:09.245679Z

This error just re-appeared. Once every couple of months, just like clockwork.

cch1 2025-06-21T00:53:06.907989Z

Seriously though, it's rare enough @ghadi, that I'm probably not going to be able to catch more details.

cch1 2025-06-21T00:56:01.970979Z

Well, I did catch the error in an alternate log target: our interior slack channel. Unfortunately that has no more detail, but at least the "double namespacing" is not an issue.

cch1 2025-05-20T00:03:46.682039Z

This "issue" is a really minor one since it seems to have been provoked by an AWS infrastructure error first:

cognitect.anomalies/anomaly.cognitect.anomalies/category
cognitect.anomalies/fault
cognitect.anomalies/anomaly.cognitect.aws.client.impl/throwable
javax.xml.stream.XMLStreamException: ParseError at [row,col]:[6,3]
Message: The element type "hr" must be terminated by the matching end-tag "</hr>".
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:652)
at clojure.data.xml.jvm.parse$pull_seq$fn__4867.invoke(parse.clj:78)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:58)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$seq__5467.invoke(core.clj:139)
at clojure.data.xml.tree$seq_tree$fn__4700.invoke(tree.clj:39)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.LazySeq.more(LazySeq.java:87)
at clojure.lang.RT.more(RT.java:722)
at clojure.core$rest__5453.invokeStatic(core.clj:73)
at clojure.core$rest__5453.invoke(core.clj:73)
at clojure.data.xml.tree$seq_tree$fn__4700$fn__4710.invoke(tree.clj:47)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:58)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$seq__5467.invoke(core.clj:139)
at clojure.data.xml.tree$seq_tree$fn__4700.invoke(tree.clj:39)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.LazySeq.more(LazySeq.java:87)
at clojure.lang.RT.more(RT.java:722)
at clojure.core$rest__5453.invokeStatic(core.clj:73)
at clojure.core$rest__5453.invoke(core.clj:73)
at clojure.data.xml.tree$seq_tree$fn__4700$fn__4710.invoke(tree.clj:47)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:58)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$seq__5467.invoke(core.clj:139)
at clojure.data.xml.tree$seq_tree$fn__4700.invoke(tree.clj:39)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.LazySeq.first(LazySeq.java:73)
at clojure.lang.RT.first(RT.java:692)
at clojure.core$first__5449.invokeStatic(core.clj:55)
at clojure.core$first__5449.invoke(core.clj:55)
at clojure.data.xml.tree$seq_tree$fn__4700$fn__4708.invoke(tree.clj:46)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$filter$fn__5962.invoke(core.clj:2826)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.countFrom(RT.java:653)
at clojure.lang.RT.count(RT.java:643)
at cognitect.aws.util$xml__GT_map.invokeStatic(util.clj:175)
at cognitect.aws.util$xml__GT_map.invoke(util.clj:171)
at cognitect.aws.util$xml__GT_map.invokeStatic(util.clj:181)
at cognitect.aws.util$xml__GT_map.invoke(util.clj:171)
at cognitect.aws.protocols$parse_encoded_string.invokeStatic(protocols.clj:110)
at cognitect.aws.protocols$parse_encoded_string.invoke(protocols.clj:102)
at cognitect.aws.protocols$parse_http_error_response$fn__6120.invoke(protocols.clj:117)
at clojure.core$update.invokeStatic(core.clj:6232)
at clojure.core$update.invoke(core.clj:6224)
at cognitect.aws.protocols$parse_http_error_response.invokeStatic(protocols.clj:117)
at cognitect.aws.protocols$parse_http_error_response.invoke(protocols.clj:113)
at cognitect.aws.client.impl$handle_http_response.invokeStatic(impl.clj:31)
at cognitect.aws.client.impl$handle_http_response.invoke(impl.clj:24)
at cognitect.aws.client.impl$send_request$fn__6454$state_machine__5750__auto____6483$fn__6486.invoke(impl.clj:87)
at cognitect.aws.client.impl$send_request$fn__6454$state_machine__5750__auto____6483.invoke(impl.clj:83)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:971)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:976)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:974)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__5768.invoke(ioc_macros.clj:985)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__589$fn__590.invoke(channels.clj:100)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at clojure.core.async.impl.concurrent$counted_thread_factory$reify__492$fn__493.invoke(concurrent.clj:29)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:833)

cch1 2025-05-20T00:04:40.747369Z

A quick summary of my reading: AWS infrastructure returned an error to an SSM GetParameters request. The aws-api tried to parse the error body as XML but it was in fact HTML.

ghadi 2025-05-20T00:04:48.401139Z

probably parsing html as xml

cch1 2025-05-20T00:04:52.114059Z

LOL.

cch1 2025-05-20T00:04:53.866589Z

Yep.

ghadi 2025-05-20T00:04:56.574299Z

grab the meta off the response

ghadi 2025-05-20T00:05:24.776239Z

it has the raw stuff there, unless some InputStream that is consumed

cch1 2025-05-20T00:05:31.711309Z

I've lost it, unfortunately. This was a little burp at 1am for a service that we probably query 1000x per day for years. First time I've every seen it.

ghadi 2025-05-20T00:05:39.948449Z

👍

cch1 2025-05-20T00:06:22.253779Z

The app logged it (via mulog) to Cloudwatch and my context is kinda limited at this point. If I ever see it in a REPL, I'll grab the metadata.

ghadi 2025-05-20T00:10:13.490779Z

what is that double namespacing on the anomalies?

ghadi 2025-05-20T00:10:20.763159Z

Not sure if phone artifact

cch1 2025-05-20T00:11:52.161899Z

The first two lines are the way our logging of events name spaces anomalies.

cch1 2025-05-20T00:12:50.870459Z

It’s actually doubled up because of the way name spaces are represented in JSON