This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-13
Channels
- # aleph (3)
- # announcements (2)
- # aws (48)
- # babashka (93)
- # beginners (101)
- # bristol-clojurians (1)
- # cider (3)
- # clj-kondo (17)
- # cljdoc (1)
- # cljsrn (3)
- # clojure (208)
- # clojure-dev (2)
- # clojure-europe (19)
- # clojure-italy (18)
- # clojure-losangeles (16)
- # clojure-nl (8)
- # clojure-spec (21)
- # clojure-sweden (8)
- # clojure-uk (57)
- # clojuredesign-podcast (6)
- # clojurescript (10)
- # code-reviews (6)
- # core-typed (1)
- # cryogen (7)
- # cursive (38)
- # datomic (34)
- # duct (13)
- # emacs (13)
- # fulcro (16)
- # funcool (2)
- # graalvm (1)
- # lambdaisland (5)
- # luminus (8)
- # lumo (1)
- # malli (2)
- # off-topic (12)
- # pathom (9)
- # re-frame (13)
- # reagent (11)
- # ring (3)
- # shadow-cljs (15)
- # sql (19)
- # tools-deps (61)
- # xtdb (23)
Why would :Body be nil on a S3 GetObject request?
(aws-api/invoke
s3-client
{:op :GetObject
:request {:Bucket billing-s3-bucket
:Key "hourly-cur/hourly-cur/20200201-20200301/fc247861-fb77-460c-9f91/hourly-cur-7.csv.gz"}})
=>
{:LastModified #inst"2020-02-04T09:04:14.000-00:00",
:ETag "\"d41d8cd98f00b204e9800998ecf8427e\"",
:Metadata {},
:ServerSideEncryption "AES256",
:ContentLength 0,
:ContentType "application/octet-stream",
:AcceptRanges "bytes",
:Body nil}
@kenny you can have keys that have no associated content. That :ContentLength
is 0 suggests that is the case, in which case s3 sends back no body. You can see this if you eval (:http-response (meta *1))
(assuming your last exp was the invoke call you posted).
I believe there is an issue with S3 returning a key that looks like this //
. If you create a CUR with no prefix set, AWS will store the CURs under the path //<report name>/<date range>/<assembly id>/<report item>
. If I call :ListObjectsV2
on this bucket, I get a list of objects returned but the :Key
is not correct. For example, one key returned looks like this "/test/20190501-20190601/7c0795d3-9b7e-438b-bb5d-de3ead37baad/test-1.csv.gz"
when its actual key should be "//test/20190501-20190601/7c0795c3-9b7e-437b-bb5d-de3ead37baad/test-1.csv.gz"
.
If I call :GetObject
on the key returned by aws-api, I get a NoSuchKey anomaly. If I try using the actual key with the //
, I get a forbidden anomaly:
{:Error {:HostIdAttrs {},
:StringToSignBytes "...",
:CanonicalRequestBytes "...",
:CanonicalRequestAttrs {},
:Message "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
:StringToSign "AWS4-HMAC-SHA256
20200213T160537Z
20200213/us-west-2/s3/aws4_request
...",
:CodeAttrs {},
:RequestIdAttrs {},
:SignatureProvidedAttrs {},
:HostId "...",
:StringToSignBytesAttrs {},
:MessageAttrs {},
:RequestId "E92073AA791F0EB6",
:Code "SignatureDoesNotMatch",
:SignatureProvided "...",
:AWSAccessKeyIdAttrs {},
:CanonicalRequest "GET
/compute-cost-reports//test/20190501-20190601/7c0795c3-9b7e-437b-bb5d-de3ead37baad/test-1.csv.gz
host:
x-amz-content-sha256:...
x-amz-date:20200213T160537Z
host;x-amz-content-sha256;x-amz-date
...",
:CanonicalRequestBytesAttrs {},
:StringToSignAttrs {},
:AWSAccessKeyId "..."},
:ErrorAttrs {},
:cognitect.anomalies/category :cognitect.anomalies/forbidden}
The object does actually exist under that path -- I can see it in the S3 console.What happens if you do the same with the aws cli?
Listing:
aws s3 ls
PRE /
2020-02-12 19:22:09 4 aws-programmatic-access-test-object
Get object:
aws s3 cp .
download: to ./test-Manifest.json
This is what it looks like in the console. It's definitely a strange case. However, it is the default for creating a CUR without a prefix so I'd expect this to work.
It appears to work :man-shrugging::skin-tone-2: Apparently the aws billing team thinks it's okay. It certainly seems very odd to me.
Note the aws cli is baroque & does some unspecified transforms of cmdline arguments, and is not a canonical guide
aws-api does not support keys with a path like this //test/20190501-20190601/test-Manifest.json
. The aws billing team stores CURs under that path if you don't specify a prefix.
Yes. There's 2 keys there. One that is /
and one that is aws-programmatic-access-test-object
This is what it looks like in the S3 console folders list. The first time I saw this, I didn't even think it was a folder.
Curious what the potential solution is if that's the problem. Not use http://java.net.URI?
I'm sure that was chosen for a reason. I'm pretty sure I added the (str/replace #"//+" "/")
when I was refactoring some things, but didn't question the choice of URI at the time.
More context:
(-> uri
(str/replace #"//+" "/") ; (URI.) throws Exception on '//'.
(str/replace #"\s" "%20"); (URI.) throws Exception on space.
(URI.)
(.normalize)
(.getPath)
(uri-encode "/"))
So it looks like we're using URI to avoid having to do the work done in .normalize
FWIW, it sounds like // is a legal path https://stackoverflow.com/questions/20523318/is-a-url-with-in-the-path-section-valid
You spelled foo wrong
From https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html:
In exception to this, you do not normalize URI paths for requests to Amazon S3. For example, if you have a bucket with an object named my-object//example//photo.user, use that path. Normalizing the path to my-object/example/photo.user will cause the request to fail. For more information, see Task 1: Create a Canonical Request in the Amazon Simple Storage Service API Reference.
We don't make that exception.
We do need to do it everywhere else, however.