Fork me on GitHub
#holy-lambda
<
2022-05-04
>
Nick Stares14:05:24

Heya, I want to include the chromedriver binary inside my docker image for one of my HL functions, however trying to install it I see that yum is not available. I guess it's not included with openjdk:latest ? How can I install packages inside my HL dockerfile?

Karol Wójcik15:05:00

OpenJDK latest is based on Debian. See https://hub.docker.com/_/openjdk

Nick Stares15:05:08

weird, apt wasn't working either

Karol Wójcik17:05:58

You can switch to different image that contains JDK.

👍 1
Nick Stares16:05:26

How does HL/lambda read the PATH at runtime? I have now have the binary at /usr/bin/chromedriver in the docker image and I can confirm that at runtime (using sam local invoke) by printing the file with . But for some reason etaoin can't find the executable at that path. I also confirmed here https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime that /usr/bin is on the PATH by default.

Karol Wójcik17:05:07

This has nothing to do with HL.

Karol Wójcik17:05:46

HL is a custom runtime with some additional tasks to speed up the development.

Karol Wójcik17:05:49

Let me check etaoin source code.

Karol Wójcik17:05:55

Maybe before this lets start from the very begining. Could you please do which chromedriver before running the actual jar, so that we can confirm the executable is findable in a shell?

Karol Wójcik17:05:09

You should be able to set a path to a binary via set-binary function. See here: https://github.com/igrishaev/etaoin/blob/master/src/etaoin/driver.clj

Nick Stares20:05:24

Thanks so much for helping with this even though it's not a HL problem!

Nick Stares20:05:40

Step 4/5 : RUN which chromedriver
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in e6952502ba83
/usr/bin/chromedriver
So that's not the issue

Nick Stares20:05:09

As I understand that multimethod gets called by the path flag e.g. (def driver (chrome {:path "/usr/bin/chromedriver"})) which I tried, https://github.com/igrishaev/etaoin#additional-parameters

Nick Stares20:05:38

let me try calling it directly

Nick Stares21:05:08

(ed/set-binary (chrome-headless) "/usr/bin/chromedriver")
Does not work

Karol Wójcik03:05:52

What is a little bit suprising for me is that you are on arm64 and you’re using amd64 image.

Karol Wójcik03:05:06

Maybe an issue is not exactly there but I found it surprising. Could you please provide a minimal repro for this issue? I'm happy to help with this.

Nick Stares15:05:24

I was able to reproduce it with clojure -Tclj-new create :template holy-lambda :name company/example-lambda :output holy-lambda-example

nick@Nicks-MacBook-Pro-2 holy-lambda-example % sam build
Your template contains a resource with logical ID "ServerlessHttpApi", which is a reserved logical ID in AWS SAM. It could result in unexpected behaviors and is not recommended.
Building codeuri: /Users/nick/code/holy-lambda-example runtime: None metadata: {'Dockerfile': 'Dockerfile', 'DockerContext': '/Users/nick/code/holy-lambda-example', 'DockerTag': 'v1'} architecture: x86_64 functions: ['ExampleLambdaFunction']
Building image for ExampleLambdaFunction function
Setting DockerBuildArgs: {} for ExampleLambdaFunction function
Step 1/4 : FROM openjdk:latest
 ---> 15bbbf63e09d
Step 2/4 : MAINTAINER Karol Wójcik <[email protected]>
 ---> Using cache
 ---> e1984a62bbbc
Step 3/4 : ADD .holy-lambda/build/output.jar output.jar
 ---> 149038560520
Step 4/4 : CMD java -jar output.jar "company.example-lambda.core.ExampleLambda"
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in d99c5540dc58
 ---> 4379543223d4
Successfully built 4379543223d4
Successfully tagged examplelambdafunction:v1

Nick Stares15:05:14

By the way, etaoin was throwing a binary not found error but when I dug deeper it was finding the binary, just not able to call it because of the older jdk I was using, https://bugs.openjdk.java.net/browse/JDK-8212828, when I upgraded the jdk it stopped being an issue

Karol Wójcik15:05:08

Cool, i’m very happy you found an answer for your question.

👍 1
Nick Stares15:05:52

What other info about my system would be relevant for helping you debug the Warning?

Karol Wójcik15:05:14

I don’t need any more info.

Karol Wójcik15:05:34

It all depends on which architecture you do want to target.

Karol Wójcik15:05:27

If you’re targeting Intel and you’re on Mac M1, then you will experience this warning.

Karol Wójcik15:05:41

It's because docker is emulated via Rosetta.

Nick Stares15:05:07

Ah, well that's what I'm doing. Yeah what I'm targeting doesn't make much difference to me, I don't have any pressing perf requirements

Karol Wójcik15:05:50

Ok, got it. Therefore you have to live with this warning.

👍 1
Nick Stares15:05:21

Hey different question, I now have 4 different lambdas in one repo that share code but also have different dependencies. Because it's all being built together, my docker image is now 1.2GB, which has to be uploaded 4 times, even if I only want to make a change to one lambda. What are my options for separating them out?

Nick Stares15:05:07

They're currently all in the same stack but I don't think that's a requirement for me

Nick Stares15:05:55

My motivation for separating them is mainly the slow deploy times, especially when I have poor upload speeds 😉

Karol Wójcik15:05:01

Cannot you build the docker image one time with all the dependencies and point to different lambdas via custom CMD?

Karol Wójcik15:05:28

Have you also considered trimming you docker images? Maybe by using Alpine images?

Nick Stares16:05:00

Hmm, interesting, I could try that. 3 of them are using the same base image but the 4th is has a different base image, but I could potentially narrow it down to 2 Dockerfiles with multiple CMD targeting different entrypoints?

Karol Wójcik16:05:35

Or maybe even one if it's possible.

Nick Stares16:05:01

Cool, I will give that a shot 🙂 And yes, I will try using trimmed images, and maybe also see if I have any unnecessary deps at this point

❤️ 1
Nick Stares11:05:05

As a test, I removed one of the functions with a large image from the template.yml. Rebuilt fine, yet sam deploy still tries to push the image (there should only be 3 layers)

nick@Nicks-MacBook-Pro-2 amc-utility % sam deploy
42f1d7875f37: Pushing [>                                                  ]  6.652MB/1.57GB
dc1b24232ab1: Layer already exists
f1ddd4093f07: Layer already exists
e57f007acf74: Layer already exists
Could it be stuck in a cache somewhere? I am not passing any cache flags.

Nick Stares11:05:50

Also, when the deploy succeeds, it deletes the function from the stack. But on subsequent deploys, it still tries to push the image for the deleted function, but "underneath" each other function!

Karol Wójcik11:05:40

I'm not an AWS SAM expert.

Nick Stares11:05:02

Ok, sorry to bug you about this, maybe I should post in a SAM forum.

Karol Wójcik11:05:31

But I would say you probably have something like samconfig.yaml or packaged.yaml that causes this.

👍 1