Re: AWS S3 big files. I've seen multi gigabyte objects uploads, so single large files can be done, but the size of the object needs to be known beforehand, which might be a reason that libs tend to keep the data in memory

for multipart uploads, the size of the part needs probably be known beforehand

the Java libs have a nice TransferManager, that can do parallel multipart downloads, if the object was uploaded in multipart fashion

would be need to add such support for aws-api too 🙂

If I remember correclty current aws-api wants to keep the whole file in memory at least when downloading. http client used by aws-api does not support streaming. amazon java sdk do not have that limitation.

what would go around all this big file stuff would be support for pre-signed upload requests in the aws-api client. I’m hoping to see this soon

the bring-your-own http client will help :)

@viesti @jsyrjala multipart uploads are already supported. Call the various Multipart operations with your input split into chunks downloading large files is problematic because of no streaming

CreateMultipartUpload -> UploadPart (many, you can do it concurrently) -> CompleteMultipartUpload

ah, true

have used only multipart download via the java libs in the past, since had a Redshift cluster write data to S3 in parallel :)

there is no multipart download

yup

you can do byte range requests on S3 objects in parallel, though. Similar effect 🙂

got carried away remembering old project :)

nice point

we've thought about some "userspace helpers" for aws-api, like paginators, etc. but so far we're focusing on the raw operations

(presigning URLs is in that ballpark too)

if there would be 3rd psrty "userspace" libs, would it be ok to use aws-api as name prefix?

I can't stop you, but viesti.aws-api would be better 🙂

:D

liking that already :)

would have to do some explaining on that particular name though :)