How many files are you downloading in parallel? And how exactly you are doing that?

we have multiple users downloading or uploading. so there is a constant flow of requests. also there is option to select multiple files and download. in that case, we spawn multiple thread and download in parallel

We use claypoole's pmap like this and I haven't seen this problem yet:

(cp/pmap 100 #(s3/get-object bucket-name %) keys)

I guess your files are probably much bigger than ours (they are quite small, a few kilobytes usually)

Do you know how many of them are running in parallel when the problem happens? If it's more than 100, I would think about introducing a queue and delay the new requests, because I guess you won't get better throughput by launching that many connections from a single machine. https://aws.amazon.com/blogs/developer/enabling-metrics-with-the-aws-sdk-for-java/ might be useful as they recommend https://github.com/aws/aws-sdk-java/issues/2663.

yeah we have most of our files over 10Mbs. some may be around 500Mbs. Also its not just once. There will be constant hits with this ranges. Thats why we are considering increasing the connection pool size. any ideas on how to achieve it with amazonica

What I'm saying is that increasing the pool size might not be the best idea because it can already be quite big. Anyway, did you look at this? https://github.com/mcohen01/amazonica#client-configuration -> https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html

Yeah. But not sure how too add it in configuration and use it while calling s3/get-object.