Fork me on GitHub
#aws
<
2019-05-15
>
orestis07:05:44

I’d like to use SQS for a low-traffic mail-sending queue (say less than 5k emails/day), but both the producers and the consumers would not be running on AWS (yet, they will in 6 months time). Is this a crazy idea or pretty normal?

orestis12:05:15

Cool! I see that they say latency is at most low hundred ms so for an email queue that sounds acceptable.

kenny17:05:48

What are valid blob types I can pass to aws-api?

kenny17:05:19

Ah, the specs are useful 🙂

kenny17:05:21

(s/form :cognitect.aws.s3.PutObjectRequest/Body)
=>
(clojure.spec.alpha/or
 :byte-array
 clojure.core/bytes?
 :input-stream
 (clojure.core/fn [%] (clojure.core/instance? java.io.InputStream %)))

ghadi17:05:55

we might expand that in the future

ghadi17:05:09

specifically to handle larger than memory objects

1
kenny17:05:30

If my :PutObject request fails, does aws-api automatically retry the request?

kenny17:05:32

Ah I see - it's set on the client and, optionally, on the individual request. Nice.

kenny17:05:43

Has it been considered to have the client implement Closable?

kenny17:05:43

Also could a client? function be added?

ghadi17:05:07

why the predicate?

kenny17:05:34

i.e. right now I need to do this:

(s/def ::s3-client #(satisfies? aws-client/ClientSPI %))

kenny19:05:28

Out of curiosity, does anyone know if Azure or GCP expose their APIs via data files like AWS does?

kenny19:05:26

Looks like they do provide swagger JSON files for all their APIs.

kenny19:05:17

If you could transform the AWS API JSONs into the OpenAPI format, it seems like the aws-api concept could work for multiple cloud providers.

ghadi19:05:42

there's a lot of little exceptions to the AWS APIs

kenny19:05:35

Kinda figured. Know any examples offhand?

ghadi19:05:32

almost all of s3

ghadi19:05:48

some apis require extra headers with md5 of body

ghadi19:05:09

S3 is the "weird API"

kenny19:05:11

Oh, that stuff can't be documented with openapi?

ghadi19:05:43

it's not in the amazon descriptors, so it wouldn't be in any translation of those descriptors

kenny19:05:49

How does aws-api handle it?

ghadi19:05:12

special code that runs on particular requests

ghadi19:05:29

there's an private extension point through a multimethod

kenny19:05:03

Ah. You don't think you'd be able to do the same thing with multiple cloud providers?

ghadi19:05:06

sure. I'm just trying to say that json descriptors isn't enough for at least AWS

ghadi19:05:22

there's particular API domain knowledge necessary, unfortunately

ghadi19:05:41

you may want to look at official GCP client sdks and look for these exceptional cases

ghadi19:05:01

searching for interceptors, request transformers, stuff like that

kenny19:05:10

It seems like you'd be able to handle that in a similar manner to how aws-api handles "special" apis

kenny19:05:14

Interesting. My company is adding support for additional cloud providers and we really enjoy working with aws-api. Any thoughts on how difficult it'd be to change aws-api to take in OpenAPI JSON instead of the aws-specific JSON files?

ghadi19:05:38

personally I wouldn't try to modify, but I'd take the same approach. input = descriptors -> output = functions that can assemble and disassemble request/response maps

ghadi19:05:39

then you wire the functions with an HTTP client

kenny20:05:22

Too many things coupled to aws in aws-api to try and modify it?

ghadi20:05:05

the descriptor transforming code isn't open source

kenny20:05:53

Oh really? Is that non-trivial to do?

ghadi20:05:13

but mainly I wouldn't try to modify the client because it's conceptually a simple thing

ghadi20:05:24

you get descriptors from google, and you shred them into functions

ghadi20:05:39

functions that operate on maps

ghadi20:05:51

I'm doing something similar with a native protocol buffer client

kenny20:05:08

Right. The main thing that will differ, I think, is the authentication.

ghadi20:05:19

something that takes .proto files and builds functions from maps -> bytes and bytes -> maps

kenny20:05:20

It almost seems like this version of aws-api that is cloud provider independent is simply an OpenAPI HTTP client.

ghadi20:05:50

i don't follow

kenny20:05:07

You could write a library that exposes any OpenAPI spec'ed API in the same format that aws-api takes.