Fork me on GitHub
#aws-lambda
<
2023-02-08
>
pesterhazy16:02:46

We're running into file size limits when loading an uberjar in Lambda. We're basically at a point where adding a new dependency puts us over the limit. According to the docs the limit is 250M but that doesn't match our uberjar size, which went from 100M to 114M. Yes it's big, but not as big as they say is allowed

pesterhazy16:02:26

Any idea on how to navigate this? It can't be that the jar is simply full right? :-)

ghadi16:02:27

lambdas support docker images with larger limits now

👍 2
pesterhazy17:02:29

I was hoping not having to make that jump but if all else fails that's definitely an option

jmglov17:02:23

Splitting the deps out into a separate layer might be an option as well.

pesterhazy17:02:08

Alas, size is total of all layers so adding layers doesn't help with the 250M limit

😭 2
ghadi18:02:01

the 250M limit is about unzipped

pesterhazy18:02:38

Yeah it's what you get when you unzip the jar

pesterhazy18:02:41

unzip -Zt uberjar.jar

pesterhazy18:02:11

Took me a long time to grok that du -sh doesn't show the right result

ghadi18:02:35

they are trying to encourage you to use container images with their asymmetrical limits

pesterhazy18:02:05

Yeah I'm letting go of my resistance to docker after 10 yrs

pesterhazy18:02:38

I thought I could wait this one out but I was wrong about that

ghadi18:02:44

i agree that docker is pointless for java

viesti13:10:33

re-found this channel, and this thread 🙂 kind of had this 250MB unzipped in my mind, when trying to add a layer to our a lambda (AWS's own enhanced monitoring layer), that was ~180MB unzipped, but went over the 250MB limit we are using Snapstart, and before that native-image, but would not like to go back to native-image and plain jvm startup is kind of slow (we currently have kind of traditional ring app as backend, just run in lambda, maybe later figuring out our infra budget better, if we could run the app just via fargate :)) Snapstart has been working for us, but our use is quite infrequent, and had a chat with AWS support, that Snapstart works best for frequently invoked Lambdas, because checkpoint cache is warm, so we have seen outlier restore durations in the order of tens of seconds, while usually around 500ms-1s, IIRC Haven't done it yet, but was thinking of trying proguard at some point, since that would probably, in our case, shave off quite a bit from the jar size, might help in getting less stuff pulled into snapstart checkpoint caches too. I was in one project where we obfuscated with proguard a Clojure app, there were some quirks IIRC (instead of :import, use fully qualified symbols etc.) but it worked kind of ok, so might try that again

viesti13:10:02

I haven't looked into yet, but it occurred to me that docker has this https://aws.amazon.com/about-aws/whats-new/2022/09/introducing-seekable-oci-lazy-loading-container-images/ and there is a paper, which I haven't also read yet, about how Lambda does image content loading https://arxiv.org/pdf/2305.13162.pdf

viesti13:10:22

apparently, won a best paper award at usenix https://twitter.com/Werner/status/1686403851308404736 sorry for posting stuff to old thread 😄

David G23:02:44

@U06QSF3BK what did you end up doing? I'm having similar issues where snapstart is slow, started a journey of separating the source code to make my api lambda jar slim, but thought that there must be something else that could be done here. I see https://blog.symphonia.io/posts/2023-01-24_snapstart-how but that gets it from 4s to 1.2s which is unacceptable for an API

viesti06:02:21

Hi! I ended up adding a hourly warmup ping, since the docs say > Functions that are invoked infrequently might not experience the same performance improvements. and we haven't now for some months, seen outliers where the lambda startup takes a long time. Not sure if something changed in the Lambda infrastructure or did the hourly invocation actually help to get the snapstarted process image into a warm cache line, but anyway, haven't seen slow outliers now. I did the change some weeks before reInvent, and I remember that it took maybe a week or couple to for me to realise that there have not been slow outliers. The slow outliers were not that frequent before either, so was kind of hard to say definitively, things are better now.

viesti06:02:14

Thanks for the article also! I've seen https://dev.to/aws-builders/aws-snapstart-part-11-measuring-cold-starts-with-java-21-using-different-deployment-artifact-sizes-4g29 too where they do "priming" in the beforeCheckpoint call, and that indeed sounds a natural place to get the process more ready. I'd just hope that lambda would get "boot images", but that might look too much like EC2 😄 We have another use case where, during deployment, we initialize an EC2 machine with a docker image, shut it down, and then when needed, call EC2 API to start the instance and then call into the service that the docker image offers (NLP stuff). The snapstart thing, after reading the paper (https://arxiv.org/pdf/2305.13162.pdf), kind of looks like EBS if you squint a bit 🙂

jmglov07:02:04

@U04997TAVR7 Have you looked into provisioned concurrency?

David G18:03:02

Don't know why I missed this. Yes, I considered that, but any container booting up above the provisioned concurrency will experience the 1s cold start, to my understanding, which is unacceptable. We switched to ecs fargate containers instead @U054AT6KT

1
👍 1