Typo in the very first sentence. I would guess this should end with the word "community"? > This annual survey collects and publishes information on the Clojure.

fixed, thx

we do this in clojure itself

in https://github.com/clojure/clojure/blob/master/build.xml for direct linking - there's a Maven property directlinking that defaults to true. some of the maven goals set Java system properties with the value of the Maven property. the Java system property can then be read in the Clojure process. not sure if that covers what you need

I can add the JDK in a custom runtime and then Clojure on another layer. All comes soooo close.

But then the shared libraries complain that they come from different loaders

Isn't the JDK already part of the snapstart image?

Huh, actually it only supports Java 11 so I guess code editing has to be abandoned

> SnapStart supports the Java 11 (`java11`) https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html. Other managed runtimes (such as nodejs18.x and python3.9), https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html, and container images are not supported.

@U04V5V0V4 how does Java 11 relate to "code editing"? Would it be different for 17 or 19? I don't get it :)

They only go up to 11 :drum_with_drumsticks:

ah

You could build your own version of nbb with your libraries included, like https://github.com/logseq/nbb-logseq did (they have added datascript and some logseq stuff). Then you won't fight incompatibilities but do get scripting in the console

the code I want to run is very JVM

gotcha

you can do the same with bb, include the JVM libs you want and make a custom version that's still scriptable

it's not something I'm making from scratch

Isn't there is a capability in lambda to keep your nodes warm? Could you enable that? (naturally this may cost some money).

I'm happy to do the work if bb ends up being the only option but ... it's not likely

One thing we bumped into with lambda a few years ago was a limit on jar size was quite low (e.g. 500 MB).

they have provisioned concurrency yes but that's a bit different

I've heard of people setting up cron jobs to hit their lambdas every 30 minutes to keep them warm too.

my idea is not to ship an uberjar but to use snapstart to download the deps on first startup 😉

then the JAR is tiny

but I might still hit some other limit

Sounds like downloading the jar could slow things down then. (Even if your clojure code runs instantly).

yes but snapstart fixes that ... it takes an image of the Lambda just before the event is fired

ok

it's an experiment to be honest so that's my current understanding anyways

I assume a $3.50+ a month AWS lighsail instance isn't an option? - you'd have no cold start with that.

can those instances be used like Lambdas to react to S3 events?

Lambdas have some interesting operational and platform properties that I don't think are generally supported on all instances

Some people will create lambda's that just are one line bash/`curl` to hit their proper service.

er, ok. That wouldn't help me here. I want to post a large number of files on to S3 and have a Lambda process one or more of them in a swarm.

I could do that with other options but it's my dream so please don't wake me up 🙂

Yup - lambda has the advantage of auto scaling which is nice.

It's certainly more of a challenge converting an existing slow-start clojure/java project to lambda than starting with lambda on a greenfield project. BB is very compatible with clojure - you may find that a lot of what you need works out of the box with it. Wish you the best of luck with it!

Thanks Rupert ... yes @U04V15CAJ already mentioned bb above and I did explicitly rule that out at the top of the thread. But I might end up eating my words 🙂

Is > code is editable at the console a hard requirement? I'd argue that the DX is worse than making the changes in your REPL and then running a script to deploy the new version of the lambda, so I wouldn't necessarily rule out the JVM/docker versions

Seems like a dream, I know and of course REPL is best but once you feel the deployment times for trying out small tweaks you feel the win

I agree with the bigger point that it’s a preference more than anything

maybe by bundling the dependencies and clojure itself in a Lambda layer and the actual lambda package itself would contain only the Clojure source code and the handler class (AOT compile only the namespace containing the handler), and in the handler, use clojure.core/resolve to look up rest of the Clojure code and let clojure compiler emit and run the bytecode, like normally, and let this happen during snapstart, so snapstart then freezes a warn jvm process didn't look if it is possible to use layers with the jvm11 runtime though

now that said the above out loud, guess I have to find time to test actually 😛 I like the dream too!

are there any technical details on snapstart? I can only find high level descriptions. I mean can it successfully "resume" a clojure runtime? can't find what JVM feature it is built on specifically?

it works on the VM level afaik but there are additional Java specific features for runtime hooks here https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html

Snapstart is a Firecracker level feature, hibernates the whole process

https://docs.aws.amazon.com/lambda/latest/dg/snapstart.html > With SnapStart, Lambda initializes your function when you publish a function version. Lambda takes a http://aws.amazon.com/blogs/opensource/firecracker-open-source-secure-fast-microvm-serverless/ snapshot of the memory and disk state of the initialized https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html, encrypts the snapshot, and caches it for low-latency access. When you invoke the function version for the first time, and as the invocations scale up, Lambda resumes new execution environments from the cached snapshot instead of initializing them from scratch, improving startup latency.

A more detailed blog post is https://aws.amazon.com/blogs/compute/starting-up-faster-with-aws-lambda-snapstart/

@U06QSF3BK I think what @U054AT6KT is doing with Blambda offers a way forward - he has an experimental JRE branch

since Firecracker open source, maybe one could play with that on own computes even 😛 https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md

https://github.com/jmglov/blambda/tree/jre-backend

ah, I havent yet seen that Blambda talk, have to watch it, was there a link to recording?

neat stuff :)

https://www.youtube.com/live/NfgYon96dsE?feature=share&t=3342

thank you!

but yeah, maybe moving that aot compilation to happen at snapstart phase could be a thing to try out, to allow skicking with jvm, with a bit of lambda layer misuse

@U04V5V0V4 I think you know this, but just in case: SnapStart only works with the standard Corretto Java 11 runtime, so custom runtimes can't make use of it at all (at least for now). Your approach of shipping a lambda artifact that hotloads deps is an interesting idea, but that doesn't require a custom runtime, nor would you need to download deps during lambda initialisation. I guess you'd want a layer that includes all your deps and then create a classloader that will suck them in from /opt in your lambda artifact. Hrm... this might just work, actually.

yup, I think that is worth to try 🙂

layer content might even be on the classpath already https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path

Of course. :face_palm::skin-tone-2:

didn't try out though yet

The race is on

well dang > The code editor does not support the Java 11 (Corretto) runtime. the split into app and layer works though

0% cat deps.edn
{:paths ["src"]
 :deps {com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.1"}
        org.clojure/clojure {:mvn/version "1.11.1"}}
 :aliases
 {:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.5"
                                                :git/sha "9c738da"}
                 babashka/fs {:mvn/version "0.3.17"}}
          :ns-default build}}}
viesti@Nektariini ~/projects/clj-lambda-layer
0% cat src/layer_demo/handler.clj
(ns layer-demo.handler)

(gen-class
  :name "layer_demo.handler"
  :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler])

(defn -handleRequest [this in out ctx]
  (println "Hello!")
  (spit out "Hello!"))
viesti@Nektariini ~/projects/clj-lambda-layer
0% cat build.clj
(ns build
  (:require [clojure.tools.build.api :as b]
            [ :as io]
            [babashka.fs :as fs]))

(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def lambda-file "target/lambda.jar")

(def layer-file "target-layer/layer.zip")
(def layer-zip-dir "target-layer/layer")

(defn lambda [_]
  (b/delete {:path "target"})
  (b/copy-dir {:src-dirs ["src"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/jar {:class-dir class-dir
          :jar-file lambda-file}))

(defn layer [_]
  (b/delete {:path "target-layer"})
  (fs/create-dirs (fs/file layer-zip-dir "java" "lib"))
  (doseq [[_ {:keys [paths]}] (-> basis :libs)]
    (doseq [path paths]
      (fs/copy path (fs/file layer-zip-dir "java" "lib" (fs/file-name path)))))
  (b/zip {:src-dirs [layer-zip-dir]
          :zip-file layer-file}))
viesti@Nektariini ~/projects/clj-lambda-layer
0% clj -T:build lambda

viesti@Nektariini ~/projects/clj-lambda-layer
0% l target/lambda.jar
-rw-r--r--  1 viesti  staff   6.1K Mar  1 21:41 target/lambda.jar
viesti@Nektariini ~/projects/clj-lambda-layer
0% clj -T:build layer

viesti@Nektariini ~/projects/clj-lambda-layer
0% l target-layer/layer.zip
-rw-r--r--  1 viesti  staff   4.1M Mar  1 21:42 target-layer/layer.zip

oooh nice ... will you put it on GitHub 🙏:skin-tone-3:

yeah, realizing that diffing to slack posts has it’s limits :D

slack, a database kinda

https://github.com/viesti/clj-lambda-layered, didn’t get the speedup that I thought with snapstart though, but will investigate later

From cold Lambda still has to transfer the jars + memory state across the network to an available host - I guess that takes time too? e.g. 100 MB jar + 512 MB memory over 1gbps connection is 4.8s just for transferring before it can even beginning to startup the process.

I've got a pairing session with @U054AT6KT on Friday at 9am and we'll look it over @U06QSF3BK... also let me know if you're free 🙂

one assumes - from the tech docs - that subsequent to call 1 the image would show the speed ups

I guess, but the host might be busy dealing with other lambdas - so it may not be able to place you back on the same host each time.

that shouldn't matter

the whole idea is that it just uses the same image

ie the only cost is rehydration from SSD

But the host needs to fetch it over the network if it hasn't got it already.

I suppose a network is always involved with AWS but no more costs than any other image

low # ms

at least that's the theory they are selling

there is one gotcha that the images have an active / inactive state which changes if the Lambda is not called for 45 mins IIRC

Must be a bit of a balancing act for them. They want high utilisation of hosts, but also want to keep enough spare capacity on hosts so that lambda tend to run on the same host each time and not delayed waiting for a slot on the host to be available.

:man-shrugging::skin-tone-3:

definitely their problem 🙂

I think I cracked it https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html

> Restore duration > 192 ms > Duration > 145.07 ms

hoho - niiiiice!!

https://github.com/viesti/clj-lambda-layered/commit/6ad218cb009f9eb662675fe3fe7586bcd6c17a88

> When Lambda creates a snapshot, your initialization code can run for up to 15 minutes.

so you get a CI to use for 15mins at max

yes - that's nice isn't it 🙂

touch the prod code enough to compile it under 15min

or where the deps can be gotten

hmm, I think this counts runtime only, deps probably fetched before that

but yeah, neat hack maybe 🙂

that example has only clj-http in the layer, next try beef it up with more deps

that's not clear ... it's initialization in my book

yeah, not sure, docs said > your initialization code can run for up to 15 minutes. so thinking that at this point, the host that runs this should have procured all the code to run

anyway, have to put myself into a checkpoint now, restore tomorrow 😄 💤

Argh! Coretto 11 doesn't support the code editor! 😭

The only win here might be making a quicker to deploy lambda artifact, as we don't get console editing and I doubt anything we can do will improve SnapStart performance.

yeah, that snapshotting wasn't too snappy, but the cold start invocation was

so waiting when snapstart gets supported by other runtimes... and yeah, I guess same effect would come with a single artifact, without a split to layer, since editing not currently available

I don't get the appeal of editing directly in the console. You still have to wait for the code to be deployed after you finish, so is there really a performance benefit compared to deploying from your computer? If you isolate the deps in a layer it should be rather quick.

the upload is quick, checkpoint takes a bit of time

so could do that later even...

one thing still, in that example that I did, I left clojure->bytecode compilation to happen at checkpoint time, which makes the checkpoint a bit slower than if it would not have to do clojure->bytecode compilation, so next thought is to include in the layer, the AOT of the clojure libraries used by the lambda application I think this could be going towards incremental compilation… goal being to have fast deploy, though a bit slower initial deploy, but later deploys faster

@UEQPKG7HQ I agree that it's not a deal breaker but seeing your code there and being able to make that one small tweak is nice. Since the rest works, we can now use Clojure with Lambdas very directly and that's enough of a win for me 🙂

the logical extreme is to have a classloader layer, that sideloads new bytecode at event invocation, before event handling 😛

(Sorry @U04V5V0V4, didn't mean to derail the conversation, I only asked because my experience with AWS console has been bad for the most part (e.g. it's so frustrating that it randomly asks you to refresh the browser even if you're active on a page), so I was genuinely wondering what I might be missing out on! 🙂 I'm really interested in this too btw, we're looking into setting up Clojure lambdas to run background jobs for our main API)

then to detect somehow if the process underneath got recycled, and start with the most fresh new whole bytecode bundle

logical extremes FTW

@U06QSF3BK myself and @U054AT6KT had a play with your code and confirm that it's working a treat.

One tweak is whether you need to call the handler rather than just resolve the var in the crac hook?

ah, true

and you seem to be calling the handler where it should just be printed if I'm not wrong

anyway - great code, great docs and it all works ... so epic work that man!!

thank you :)

it was a quick hack, nice that it went into generally ok direction 🙂

haha you're way too modest

I think that next thing could be to include the AOT of library clojure code in the layer too

😄

one thing I would like is a Clojure layer that can be used to download the deps as part of the snapstart initialization

if you don't get to it first, I'll have a look at that next week

or I'll persuade @U054AT6KT .... actually I know he's already convinced 🙂

nice 😄

it would be interesting to achieve a workflow where one works locally in a repl, and every now and then syncs the application to make a new deploy, and try keep the actual lambda process start after deploy fast

yes - we were discussing exactly that. It would be nice to have the best of both worlds where you could eventually use all of the pre-downloads and pre-building but not always

I did a setup for a project that I now work in where we have Lambda as the backend and use https://github.com/FieryCod/holy-lambda-ring-adapter to code the app as ring-style handler and locally use jetty. Current prod deploy involves graalvm native-image, which takes a while to compile, but alongside there is a jvm11 runtime deploy to of the lambda, just a boolean to switch what to deploy so with snapstart + jvm, I envision that you could run locally with ring<->api-gw-style-lambda-event translation and still deploy fast, ontop a snapstart deploy

hmm, I might have confused there things a bit, might be that if you do a lambda version publish with snapstart and then upload new lambda code version, that probably doesn't use the snapstart cached version any longer

well, anyway, interesting hacks ahead still I think

Yeah, fun stuff to be sure!

Well, had to try out that sideloading…

https://github.com/viesti/clj-lambda-layered/blob/main/build.clj#L26-L30 now has AOTed code, except the app, lambda has only handler, no app code, clojure code is sideloaded and reloaded if there is a new code bundle, so the running process get’s to reload code, kind of the same way as you would locally, if you’d reload the core that you call from your event handler, idea is that copying only sources makes for less stuff to copy obviously loading code from internet is a bit fishy, but could put the bundle into a location where only the lambda instance has access (do authenticated read from s3, or say from nfs share…)

this way you get to skip lambda deploy, for loading new code, assuming that you write reloadable code

also, sideloading works at checkpoint time too, so you could do initial checkpoint, which takes a bit of time, and then continue via sideloading, to get a faster start even if the process/vm got recycled

I guess the sideloader could be behind a switch (environment variable) etc.

Holy cow - that’s spectacular!!

🙇:skin-tone-2: @U06QSF3BK

OMG I'm both very impressed and very annoyed, because I was just sitting down for a nice Sunday hacking sesh to get that AOT compilation done. 😂

On the serious tip, this is really amazing stuff! 🏆

OK, now I can refocus my hacking on building a site analyser for my blog. 😉

😄