Clojurians Log v2

from 39 failures, 12 errors to 15 failures, 12 errors by fixing test 🧌

just what I was looking for too 🙂

very cool 🙂 ! I go at the restaurant for 2 hours and you guys already did the review job ^^

>>> For some services, you must include the X-Amz-Security-Token query parameter in the canonical (signed) query string. For other services, you add the X-Amz-Security-Token parameter at the end, after you calculate the signature. For details, see the API reference documentation for that service.

Ok and how do I know?

héhé

from sign4 or 2 ?

hmm, might this relate to role based access control

so iot is an exception https://github.com/mhart/aws4/issues/30

sigv4 non s3

mkay

all tests pass

nice 🎉

🦜

can you quickly explain what went wrong ?

> Yay, all your tests have been fixed!

says CircleCI too

lol

I hope they gave a test suite for signing v2

there were errors in the tests: • treating normalize-path.txt as a test case • multi-valued headers (both multiline value and multiple occurences) were not properly handled when generating the headers map

errors in signature code: • missing percent encoding for non-ASCII • improper path normalization

ok

v2 is only required by simple db, no?

I am not sure, that’s what I am looking at right now

oh and third bullet: STS token handling depending on service name and moon phase

😂

okay so it looks like sdb is the only service that supports ONLY v2

which is a pretty good news

as it’s also a “deprecated” service

and is there any compelling reason for someone to use v2 on other services?

nope, none

hmm, did we have issue for S3 support?

thinks S3 rather support would increase usefulness a lot

it’s a special protocol not yet implemented

there are 5 of them, we’ve done 1

I have to finish the query protocol, working on it, hope to come with something in the next 2 weeks

oh, I did not see the new partitions.json ^^

no more resources/aws-sdk-core/apis/ ?

how do we generate apis then ^^ ?

And I should revive the spec split

Most apis descriptors are now part of the Java SDK. @viesti worked on that

ok

most ?

S3 is missing iirc

I’d rather source from aws-sdk-ruby for example; @viesti, your opinion?

hum, it might be more complete, haven't checked

and what about script vs dev dep?

dev time script could fetch the specs

I'm a bit surprised on this amount of corners of the internet that have aws specs here and there :)

or why don’t they have a public repo for them?

maybe aws-sdk-ruby as git submodule, to keep track of specific version

from where was the initial api specs, before aws-java-sdk-models, taken from @cgrand ? (as I recall, s3 was missing from those specs too)

I don’t remember

so there is no harmonized way of having some descriptions files

from official sdks ?

I guess com.amazonaws/aws-java-sdk-models is official source, but is missing S3

apropo, resources/aws-sig-v4-test-suite should be a dev-time resource

hmm, aws-sdk-ruby is I guess official too, since it’s under aws github organization

but haven’t yet found a single language independent place of api specs

👍

So made it such. Also using classpath for resolving path to aws-sig-v4-test-suite, since remembering that in Cursive the working directory might not be the project directory (might be that this actually was for Leiningen tasks though only).

should compare list of api specs from the ruby sdk with the maven artifact

I think a git submodule would make sense for tracking a repo with the api specs

so 2 submodules, java and ruby one

one task to compare both api description

?

note that I see only what appears to be the last version of each lib =>

I compare this to the “old” way where we had all versions co-existing

which is fine, I am just saying

it looks like there is some light this way => https://github.com/aws/aws-sdk-ruby/tree/master/apis/s3/2006-03-01

user> (count model-jar-entries)
128

kimmoko@MACTN0AFH040  ~/programming/aws-sdk-ruby/apis(master|✔)
0% ls -C1 | wc -l
     136

I guess we’d need only stuff from the ruby sdk

might be yes

so thinking that would make sense to keep only the latest version of the api spec

0% du -h -d 1 | sort -h
8.0K	./.github
 32K	./tasks
 92K	./doc-src
992K	./build_tools
 27M	./apis
 53M	./gems
 58M	./.git
139M	.

#{"alexaforbusiness" "opsworkscm" "application-autoscaling"
  "lex-models" "appstream" "secretsmanager" "connect"
  "mediastore-data" "fms" "resourcegroupstaggingapi"
  "AWSMigrationHub" "s3" "meteringmarketplace" "iotanalytics"
  "elasticloadbalancingv2" "iot-data" "kinesis-video-media"
  "autoscaling-plans" "pricing" "iot-jobs-data"
  "kinesis-video-archived-media" "acm-pca"}

found in ruby and not in java

that’s 22 items, there 8 item difference in the count, hmm…

yes I know 🙂

I might be wrong because I did a difference on a set, on directory names by excluding all the 2222-22-22 dirs

different naming, for example iot-jobs-data vs data.jobs.iot

oh, nice one lol

anyway, there are more description files in the ruby repo

do we need to generate api from versions other than the latest?

user> (def ruby-sdk-apis (set (for [file (.listFiles (java.io.File. “/Users/kimmoko/programming/aws-sdk-ruby/apis”))
                                    :let [[latest-api-dir] (sort #(compare %2 %1) (.listFiles file))
                                          json (with-open [rdr ( ( latest-api-dir “api-2.json”))]
                                                 (cheshire.core/parse-stream rdr true))]]
                                (get-in json [:metadata :serviceFullName]))))
#’user/ruby-sdk-apis
user> (def aws-java-sdk-apis (set (for [entry model-jar-entries
                                        :let [json (with-open [in (.getInputStream jar-file entry)
                                                               rdr ( in)]
                                                     (cheshire.core/parse-stream rdr true))
                                              ]]
                                    (get-in json [:metadata :serviceFullName]))))
#’user/aws-java-sdk-apis
user> (clojure.set/difference ruby-sdk-apis aws-java-sdk-apis)
#{“Amazon Kinesis Video Streams” “AWS IoT Analytics” “Amazon Simple Storage Service” “Firewall Management Service” “Amazon Kinesis Video Streams Archived Media” “Amazon Connect Service” “AWS Certificate Manager Private Certificate Authority” “AWS Secrets Manager”}
user> (count (clojure.set/difference ruby-sdk-apis aws-java-sdk-apis))
8

133 entries in latest maven model artifact: https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-models/1.11.320

no S3 though 😕

right, so we go back to sourcing from ruby

is the submodule idea any good?

was thinking that bumping the source dependency would be a bump of the submodule, would not need to write code to fetch the spec files

Sounds ok but I have no strong git opinions.

trying it out

but sleep now