Fork me on GitHub
#aws
<
2023-06-21
>
Caio Cascaes17:06:41

Hey all! Howdy? When I run an AWS CLI command to retrieve values from DynamoDB, as shown below, everything works fine. The query returns 73MB of data as expected:

aws dynamodb query --table-name a-table \
    --key-condition-expression "#p = :projectval" \
    --expression-attribute-names '{"#p":"project"}' \
    --expression-attribute-values '{":projectval":{"S":"A_PROJECT_NAME"}}'
However, when I attempt the same query in a Clojure project, using the Cognitect AWS API as follows, I encounter some issues:
(s/defn ->query :- wire.out.dynamodb/Query
  [table-name :- s/Str
   item :- (s/pred map?)]
  {:op      :Query
   :request {:TableName              table-name
             :KeyConditionExpression "#p = :projectval",
             :ExpressionAttributeNames {"#p" (str (-> item keys first name))}
             :ExpressionAttributeValues {":projectval" {:S (-> item vals first str)}}}})
The resulting query only yields 1.17MB of data, which is incomplete. Are there any additional configurations I need to adjust to retrieve the complete data set? Or is there a limitation that I should be aware of?

Alex Miller (Clojure team)17:06:22

a lot of the aws calls page and tell you that in response metadata - any chance that's happening here?

Caio Cascaes18:06:03

Let me check about it

Caio Cascaes18:06:09

I'll make a loop to scan all pagination, giving a try here

R.A. Porter18:06:22

You might want to take advantage of iteration if you’re using 1.11.

👀 1
Caio Cascaes19:06:01

Indeed, it was the pagination (`:LastEvaluatedKey`). Now it went as expected. Many thanks! I'll take a look on iteration, thanks for the tip @U01GXCWSRMW! A middle aged Clojurian here 😅

R.A. Porter19:06:52

This is the best documentation I’ve seen of it: https://www.juxt.pro/blog/new-clojure-iteration/

👀 2