Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-12-14

Channels

#aws
<
2023-12-14
>
pavlosmelissinos08:12:35

I'm running PutObject in a pmap to move 50K files to S3 and while the operation itself is fairly quick (I can see the files on S3), it seems that the process has been running for many hours after it has uploaded the files. I have no idea how to troubleshoot because I can't reproduce it locally. Has this happened to anyone? I'm not even sure if it's an issue with aws-api or pmap (I tried using plain map and claypoole, neither made a difference, so I suppose that's not it but my intuition still tells me there's something wrong here) :thinking_face:

hiredman08:12:13

You have tried different ways to do the upload and gotten the same result, have you tried not doing the upload at all? Maybe it is a completely different part of your code that is keeping the jvm running

💡 1
pavlosmelissinos10:12:50

You have tried different ways to do the upload and gotten the same result, have you tried not doing the upload at all?I had actually tried skipping the step that involves uploading the files and that solved the problem. However, that step also computes some stats and there's no logging in between... > Maybe it is a completely different part of your code that is keeping the jvm running That's it... The stats looked innocuous but they actually run ListObjectsV2 for each file (so even if each invocation is instant, we still had to run it 50K times, which is stupid but we didn't notice before - usually we have fewer than 4 files, so this scenario is extremely atypical) Anyway, thanks for the perspective!