Fork me on GitHub
#aws
<
2019-11-15
>
kulminaator14:11:50

Trying to do S3 listing from within of an aws lambda fails for me with aws/api and aws/s3

kulminaator14:11:42

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;, :cognitect.http-client/throwable #error {
 :cause java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
 :via
 [{:type java.lang.NoSuchMethodError
   :message java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
   :at [cognitect.http_client$empty_bbuf invokeStatic http_client.clj 37]}]
 :trace
 [[cognitect.http_client$empty_bbuf invokeStatic http_client.clj 37]
  [cognitect.http_client$empty_bbuf invoke http_client.clj 34]
  [cognitect.http_client$on_content$fn__10264 invoke http_client.clj 140]
  [clojure.core$update invokeStatic core.clj 6196]
  [clojure.core$update invoke core.clj 6188]
  [cognitect.http_client$on_content invokeStatic http_client.clj 139]
  [cognitect.http_client$on_content invoke http_client.clj 136]
  [clojure.lang.Atom swap Atom.java 51]

kulminaator14:11:16

;; code starting like

(let [s3-listing (aws/invoke s3 {:op :ListObjectsV2 :request
                                   {:Bucket bucket-name
                                    :Prefix files-prefix}})
        s3-files (map :Key (:Contents s3-listing))]
    (println s3-listing)

Alex Miller (Clojure team)14:11:58

and if so with what jdk

kulminaator15:11:51

Will report a little later

kulminaator16:11:01

I was compiling with leiningen on java11 openjdk, and Amazon is running java8 openjdk for java lambdas

kulminaator16:11:44

Once I convinced leiningen to run with java8 for uberjar build the problem seemed to go away

Alex Miller (Clojure team)16:11:55

yeah, that's what I would have recommended

Alex Miller (Clojure team)16:11:53

there have been some changes in the jdk that affect the compiled method invocation here. I don't remember the details but it is pretty subtle

kulminaator14:11:20

Any advice is welcome, some parts of the internet suggest a jdk version issue, aws lambda is running java8

kenny16:11:40

The aws-api readme says: > specs which are generated from the source descriptions There's no obvious documentation on how to use these. How do I access these generated specs?

kenny16:11:17

Ah found it. You're supposed to use response-spec-key.

kenny17:11:16

How come enum values are not enumerated in the generated specs?

kenny17:11:26

The actual AWS API reference docs seem to have access to enum values.

ghadi17:11:30

(s/def :cognitect.aws.kinesis/ConsumerStatus #{"DELETING" "CREATING" "ACTIVE"})
enums are there

ghadi17:11:00

@kenny can you post stuff you tried, whether you were successful or not?

kenny17:11:41

Not all of them are like that:

(s/def :cognitect.aws.workspaces/WorkspaceState string?)
Reference docs: https://docs.aws.amazon.com/workspaces/latest/api/API_Workspace.html?shortFooter=true

kenny17:11:09

> Valid Values: PENDING | AVAILABLE | IMPAIRED | UNHEALTHY | REBOOTING | STARTING | REBUILDING | RESTORING | MAINTENANCE | ADMIN_MAINTENANCE | TERMINATING | TERMINATED | SUSPENDED | UPDATING | STOPPING | STOPPED | ERROR

ghadi17:11:26

looking -- what's the mvn coordinate you're using?

kenny17:11:34

com.cognitect.aws/workspaces {:mvn/version "758.2.549.0"}

kenny17:11:51

Just tried the latest 770.2.568.0 and same thing.

kenny17:11:00

Same thing for (s/def :cognitect.aws.workspaces/ConnectionState string?)

ghadi17:11:10

investigating

4
kenny17:11:15

The generators for these specs also don't work.

(gen/generate (s/gen :cognitect.aws.workspaces.Workspace/DirectoryId))
Execution error (ExceptionInfo) at clojure.test.check.generators/fn (generators.cljc:435).
Couldn't satisfy such-that predicate after 100 tries.
That's a bit harder to solve given the spec is a regex.

ghadi17:11:55

regex specs is a known problem that @dchelimsky is working on

ghadi17:11:09

not all regexes are invertible, even if we use a library like test.chuck

ghadi17:11:41

i'm looking into why (s/def :cognitect.aws.workspaces/ConnectionState string?) is not an enum

kenny17:11:48

True. Seems like test.chuck may be good enough for most/all of the aws cases though?

ghadi17:11:35

you would think but @dchelimsky tested it and it's about 50/50 from what I recall

ghadi17:11:50

some of the regexes AWS has are truly bizarre

kenny17:11:02

Not surprised haha

kenny17:11:27

test.check doesn't work on the DirectoryId regex. Oh well.

ghadi17:11:28

@kenny it turns out that the enum excludes values that the api actually responds to

ghadi17:11:51

we found that to be the case on some APIs

kenny17:11:29

Geez. Does the enum def match the api doc reference?

ghadi17:11:44

which enum def?

kenny17:11:55

The one you looked at

kenny17:11:20

ConnectionState

ghadi17:11:27

i didn't look at a specific one, I just found out the history for the change

ghadi17:11:48

it probably matches in 95% of the cases, if I were to guess

kenny17:11:00

How do you know it doesn't match?

ghadi17:11:09

matching meaning the API documentation (non-canonical) and the service descriptor (non-canonical)

ghadi17:11:23

the truth is what the API accepts and returns

ghadi17:11:36

which might not be in sync with descriptors or documentation

ghadi17:11:55

these specs are mechanically generated from json descriptors

kenny17:11:14

The only was you'd know "the truth" is by executing API calls, right?

kenny17:11:40

So these specs have been manually overridden?

ghadi17:11:02

If I were to guess what happened: someone tried to hit an API with the library's validation turned on, and a value was rejected because it wasn't in the enum spec

ghadi17:11:08

enums are closed specs

ghadi17:11:06

(value rejected by the library's validation)

ghadi17:11:14

not by the service itself

kenny17:11:33

Interesting. A bit surprised someone else is using Clojure & WorkSpaces haha. So then you guys must be storing which specs do not match and overriding with the "open" enum -- string?.

ghadi17:11:53

i never said anything about Workspaces

ghadi17:11:35

it's a general issue we hit somewhere else, that made us re-examine how we emit specs

kenny17:11:52

So all enums are open?

ghadi18:11:07

are enums in AWS specced as string? ?

ghadi18:11:23

yeah that might be a stale artifact, I'll confirm on a more recent mvn

kenny18:11:58

So all AWS enums are supposed to be specced as string??

ghadi18:11:56

(s/def :cognitect.aws.kinesis/ConsumerStatus string?)

ghadi18:11:00

on latest ^

kenny18:11:35

I understand. It's kinda a tricky situation. I'm planning to have functions that operate over certain AWS properties. Having generators generate relevant values is important. Seems like the enum values should be worked into the generated generators.

ghadi18:11:18

understood

kenny18:11:59

If the gens for things like ConsumerStatus always generate a random string it will not hit all the code paths, making the gen test bad. For now I can do something like this for all the enum specs:

(s/def :workspace-info/State
  (s/with-gen
    :cognitect.aws.workspaces/WorkspaceState
    #(gen/one-of [(s/gen #{"PENDING"
                           "AVAILABLE"
                           "IMPAIRED"
                           "UNHEALTHY"
                           "REBOOTING"
                           "STARTING"
                           "REBUILDING"
                           "RESTORING"
                           "MAINTENANCE"
                           "ADMIN_MAINTENANCE"
                           "TERMINATING"
                           "TERMINATED"
                           "SUSPENDED"
                           "UPDATING"
                           "STOPPING"
                           "STOPPED"
                           "ERROR"})
                  (s/gen string?)])))
It'd be nice to have something like that build in.

dchelimsky18:11:00

@kenny side note: you'll get better feedback (s/describe and s/explain) if you use the s/spec macro with :gen than the s/with-gen function.

kenny21:11:06

Weird. That doesn't seem like the intended use for s/spec.

kenny18:11:53

Also would be great to have a note in the readme or, even better, next to the enum specs noting why it is specced as an open enum.

ghadi18:11:56

can you assist us by capturing a github issue with details?

ghadi18:11:20

the concerns are documentation/discovery of enums, and associated gen of enum specs