Clojurians Log v2

@dainius.jocas Could it be that the slowness with curl was with repeated execution of requests? Maybe clj-http-lite uses connection pooling?

I doubt that the slowness is because connection pooling. As it is written in the README No persistent connection support

I have a suspicion that it might be because doing shell out on a very constrained environment, just 128MB RAM and very little CPU. I would like to test with more RAM given to the lambda and some profiling.

I have an idea to package babashka as a Lambda layer, and make a proper template that uses handler option, that would be a script file. In this way one could write a lambda function right in the cloud IDE or the packaging script could produce --uberscript for compactness. @borkdude what do you think if I publish babashka as a public lambda layer?

👍

@dainius.jocas Btw, if you want to develop babashka.curl you can load it with bb from the classpath with :reload, instead of the built-in one.

OK, thanks. I'll await your results.

I think it might also be good to post an issue about this to oracle/graal, since it seems they intended to fix the issue

instead of us working around it

commented here now: https://github.com/oracle/graal/issues/841#issuecomment-602181005

I've tried with try/catch. The same failure.

ok, then our decision was good

It turns out that lambda runtime has curl. So no external dependencies needed. Then the issue is to parse curl response with headers. @borkdude do you plan to support parsing curl response into {:headers {} :body "" :ect :etc} in babashka.curl?

@dainius.jocas Cool. I think it's good to have support for this, but it's not currently implemented.

Another issue with using curl is that it is much much slower, than clj-http-lite. The echo took 200ms consistently to execute while clj-http-lite take 15ms. Here is my naive implementation of curl response parsing https://gist.github.com/dainiusjocas/d2a5ca60ae25cd260f37b7c125a0b2e6

why would curl be slower you think?

I made an issue for parsing the headers here: https://github.com/borkdude/babashka.curl/issues/5 I think this should be optional, so maybe when passed an option like :response true will return the full response map and when it's not set, it will return only the body?

no idea why it could be that much slower.

IMO, the current functionality should not change, so this response parsing should under some new param like :response true. Note that response headers are retrieved with :raw-args ["-i"]

yeah, so when :response true, use the "-i" argument implicitly

👍

FWIW I see similar times between babashka.curl and slurp

user=> (time (do (curl/get "") nil))
"Elapsed time: 75.406528 msecs"
nil
user=> (time (do (slurp "") nil))

Cool! How is the cold startup time?

Don't know, ask Dainius I guess 🙂 @dainius.jocas

normally bb starts within 13 ms on a laptop

I’ll give it a spin once I’m back on computer! Startup-time is usually not that critical to me but I’m just curious.

he did say to me that there was one issue in bb that prevented this from working, he's going to do a PR for that

Prevented what from working?

https://twitter.com/dainius_jocas/status/1241329505635090432

this is the lambda example

but he didn't tell me what, so I have no clue

we will know soon hopefully

I just tried and was able to deploy it to lambda. Had to make small modofikations to makefile on MacOS because it wasn’t happy with /tmp/....

the first call took 486.57 ms which is fast imo.

And second cold call was 484.16 ms. So we could say it’s < 500ms 🙂

Hi, yeah, the cold start is 400ms, subsequent calls are 30ms. Worth mentioning is that the cold start and the duration depends on how much RAM is dedicated to the lambda, the more the faster lambda is.

Yep I saw variance between 4ms and 120ms on hot calls with 128Mb RAM. Which I think is quite good.

@dainius.jocas About your PR, what happens without it on AWS? Do you get an error? https://github.com/borkdude/babashka/pull/305/files#diff-b4e1e6a8c0b365c3b9fac4c30360f72d

Without the change Lambda fails in a very similar way as is described here https://github.com/oracle/graal/issues/841

Also, I've found very similar issue https://github.com/quarkusio/quarkus/issues/4262. Here it is said that

You can work around the problem by defining the environment variable DISABLE_SIGNAL_HANDLERS to any value. Obviously this means that you won't have OS signal handling though.

@dainius.jocas That seems to be project specific: https://github.com/quarkusio/quarkus/blob/bb1ef1ffd952d5f3a1d75e599ef5acf107e6de55/core/runtime/src/main/java/io/quarkus/runtime/Application.java#L29

It is. I'm just not sure on how to name the environment variable. But my overall solution is just a quick and dirty hack to make the thing work. Probably more sophisticated design is needed to support Lambda use case.

So far babashka specific env variables starts with BABASHKA_, like BABASHKA_CLASSPATH. What about BABASHKA_DISABLE_PIPE_HANDLER?

Pipe handling doesn't work on Windows either

ok, I'll rename the env variable in the PR

merged. I assume you don't need a new build? btw, @valtteri - for you it did work right, without this patch?

A new build would be nice because in the lambda archive I'm packaging the bb binary. For the example lambda I've deployed a babashka docker to Docker Hub https://hub.docker.com/r/dainiusjocas/babashka and then used it to build lambda archive. So, if babashka would provide a build, I'd adjust the example. Propably, later I'd convert it into a template project.

There are pre-release builds in #babashka-circleci-builds - I'll hope to release a new one soon.

Perfect! No worries, I'm in no rush 🙂

@borkdude correct, it worked without the patch

@valtteri Hmm, so maybe the patch isn't needed then - or is there something different in your env @dainius.jocas

The patch is needed, I've packaged the patched version of bb to a docker, and then used the patched bb for building the lambda in the docker.

FROM dainiusjocas/babashka:latest as BABASHKA

FROM clojure:tools-deps-alpine as BUILDER
RUN apk add --no-cache zip
WORKDIR /var/task

COPY --from=BABASHKA /usr/local/bin/bb bb

ENV GITLIBS=".gitlibs/"
COPY lambda/bootstrap bootstrap

COPY deps.edn deps.edn

RUN clojure -Sdeps '{:mvn/local-repo "./.m2/repository"}' -Spath > cp
COPY src/ src/
COPY resources/ resources/

RUN zip -q -r function.zip bb cp bootstrap .gitlibs/ .m2/ src/ resources/ deps.edn

@dainius.jocas I wonder if a simple try/catch would have also sufficed as a workaround to this

Or was it more like a hard unrecoverable crash?

if try/catch around handle-pipe! works, then that would be preferable I think

also, this commit ought to have fixed the problem: https://github.com/dmlloyd/graal/commit/dedcc86471b0209eeebf1df816b5f2590dc3361d which version of graalvm did you use to compile your docker image?

I haven't tried try/catch. Here is the error message from logs https://gist.github.com/dainiusjocas/feafeef5653ff2c6e8c7b2d9627a831d

I've used the same version as is specified in the Dockerfile

Looking at that log, I suspect try/catch won't do it, but you could give it a try if it's not too much to ask. I did release v0.0.78 now and renamed the variable to BABASHKA_DISABLE_PIPE_SIGNAL_HANDLER

Tomorrow I'll try the try/catch approach. I also suspect that it will not work because lambda runtime panics when lambda tries to do anything with signals. But I'll try