Fork me on GitHub
#aws
<
2020-07-08
>
kenny16:07:02

When using aws-api, does :PutObject end up bringing :Body fully into memory?

ghadi16:07:19

currently

kenny16:07:37

Hmm, yeah I thought so. Getting OOMs. Any workarounds?

ghadi16:07:01

multipart uploads

ghadi16:07:04

are you uploading from a File source?

kenny16:07:25

Is there an existing example for doing that? If not, will figure it out.

ghadi16:07:43

there is not an existing example

ghadi16:07:02

there's an API to start an upload, then to upload pieces, then to complete the upload

ghadi16:07:20

pieces parts can be uploaded concurrently

kenny16:07:11

Somewhat amusingly, our system uses the -XX:+HeapDumpOnOutOfMemoryError jvm flag. When the app reboots, it will upload the heap dump to s3. The heap dumps are so large that they oom the app again while uploading 😆

ghadi16:07:17

heh, whoops

ashnur17:07:34

same feature implemented twice? 🙂

kenny23:07:00

For those interested in an example for a multipart upload...