Hey all! Howdy? When I run an AWS CLI command to retrieve values from DynamoDB, as shown below, everything works fine. The query returns 73MB of data as expected:
aws dynamodb query --table-name a-table \
--key-condition-expression "#p = :projectval" \
--expression-attribute-names '{"#p":"project"}' \
--expression-attribute-values '{":projectval":{"S":"A_PROJECT_NAME"}}'
However, when I attempt the same query in a Clojure project, using the Cognitect AWS API as follows, I encounter some issues:
(s/defn ->query :- wire.out.dynamodb/Query
[table-name :- s/Str
item :- (s/pred map?)]
{:op :Query
:request {:TableName table-name
:KeyConditionExpression "#p = :projectval",
:ExpressionAttributeNames {"#p" (str (-> item keys first name))}
:ExpressionAttributeValues {":projectval" {:S (-> item vals first str)}}}})
The resulting query only yields 1.17MB of data, which is incomplete.
Are there any additional configurations I need to adjust to retrieve the complete data set? Or is there a limitation that I should be aware of?a lot of the aws calls page and tell you that in response metadata - any chance that's happening here?
Let me check about it
I'll make a loop to scan all pagination, giving a try here
You might want to take advantage of iteration if youโre using 1.11.
Indeed, it was the pagination (`:LastEvaluatedKey`). Now it went as expected. Many thanks!
I'll take a look on iteration, thanks for the tip @coyotesqrl! A middle aged Clojurian here ๐
This is the best documentation Iโve seen of it: https://www.juxt.pro/blog/new-clojure-iteration/