Clojurians Log v2

Splitting the deps out into a separate layer might be an option as well.

Alas, size is total of all layers so adding layers doesn't help with the 250M limit

the 250M limit is about unzipped

Yeah it's what you get when you unzip the jar

unzip -Zt uberjar.jar

Took me a long time to grok that du -sh doesn't show the right result

they are trying to encourage you to use container images with their asymmetrical limits

Yeah I'm letting go of my resistance to docker after 10 yrs

I thought I could wait this one out but I was wrong about that

i agree that docker is pointless for java

mostly

@viesti what did you end up doing? I'm having similar issues where snapstart is slow, started a journey of separating the source code to make my api lambda jar slim, but thought that there must be something else that could be done here. I see https://blog.symphonia.io/posts/2023-01-24_snapstart-how but that gets it from 4s to 1.2s which is unacceptable for an API

Hi! I ended up adding a hourly warmup ping, since the docs say > Functions that are invoked infrequently might not experience the same performance improvements. and we haven't now for some months, seen outliers where the lambda startup takes a long time. Not sure if something changed in the Lambda infrastructure or did the hourly invocation actually help to get the snapstarted process image into a warm cache line, but anyway, haven't seen slow outliers now. I did the change some weeks before reInvent, and I remember that it took maybe a week or couple to for me to realise that there have not been slow outliers. The slow outliers were not that frequent before either, so was kind of hard to say definitively, things are better now.

Thanks for the article also! I've seen https://dev.to/aws-builders/aws-snapstart-part-11-measuring-cold-starts-with-java-21-using-different-deployment-artifact-sizes-4g29 too where they do "priming" in the beforeCheckpoint call, and that indeed sounds a natural place to get the process more ready. I'd just hope that lambda would get "boot images", but that might look too much like EC2 😄 We have another use case where, during deployment, we initialize an EC2 machine with a docker image, shut it down, and then when needed, call EC2 API to start the instance and then call into the service that the docker image offers (NLP stuff). The snapstart thing, after reading the paper (https://arxiv.org/pdf/2305.13162.pdf), kind of looks like EBS if you squint a bit 🙂

@clojurians.slack.cee0 Have you looked into provisioned concurrency?

Don't know why I missed this. Yes, I considered that, but any container booting up above the provisioned concurrency will experience the 1s cold start, to my understanding, which is unacceptable. We switched to ecs fargate containers instead @jmglov

re-found this channel, and this thread 🙂 kind of had this 250MB unzipped in my mind, when trying to add a layer to our a lambda (AWS's own enhanced monitoring layer), that was ~180MB unzipped, but went over the 250MB limit we are using Snapstart, and before that native-image, but would not like to go back to native-image and plain jvm startup is kind of slow (we currently have kind of traditional ring app as backend, just run in lambda, maybe later figuring out our infra budget better, if we could run the app just via fargate :)) Snapstart has been working for us, but our use is quite infrequent, and had a chat with AWS support, that Snapstart works best for frequently invoked Lambdas, because checkpoint cache is warm, so we have seen outlier restore durations in the order of tens of seconds, while usually around 500ms-1s, IIRC Haven't done it yet, but was thinking of trying proguard at some point, since that would probably, in our case, shave off quite a bit from the jar size, might help in getting less stuff pulled into snapstart checkpoint caches too. I was in one project where we obfuscated with proguard a Clojure app, there were some quirks IIRC (instead of :import, use fully qualified symbols etc.) but it worked kind of ok, so might try that again

I haven't looked into yet, but it occurred to me that docker has this https://aws.amazon.com/about-aws/whats-new/2022/09/introducing-seekable-oci-lazy-loading-container-images/ and there is a paper, which I haven't also read yet, about how Lambda does image content loading https://arxiv.org/pdf/2305.13162.pdf

apparently, won a best paper award at usenix https://twitter.com/Werner/status/1686403851308404736 sorry for posting stuff to old thread 😄

Any idea on how to navigate this? It can't be that the jar is simply full right? :-)

lambdas support docker images with larger limits now

I was hoping not having to make that jump but if all else fails that's definitely an option