Fork me on GitHub
Nick Stares17:02:08

Hello 🙂 I am new to Lambda and holy-lambda and trying to setup a deps layer for my babashka backend. I added a file download-pods.clj , separate from my core.cljc :

   [babashka.pods :as pods]))

(pods/load-pod 'org.babashka/aws "0.1.2")
(pods/load-pod 'org.babashka/buddy "0.0.1")
(pods/load-pod 'org.babashka/postgresql "0.1.0")
But I'm not sure about how to go about setting up the layer

Karol Wójcik18:02:02

Do you mean a layer of dependencies?

Karol Wójcik18:02:30

1. Put the symbol/version pairs to bb.edn configuration in :pods keyword. 2. Run bb hl:babashka:sync to download the pods to a directory 3. Make a layer the same way as it's specified in documentation (babashka holy lambda tutorial), but point to .holy-lambda/pods 4. Link the layer with your function by a reference. 5. Verify if it works by using sam local invoke.

👍 1
Karol Wójcik18:02:36

holy-lambda has it's own mechanism of downloading the pods, so the pods will never get redownloaded in Lambda environment.

Karol Wójcik18:02:11

Also check if your dependencies are not bigger than 250mb, since otherwise you would have to pack the lambda in docker image.

Nick Stares18:02:32

Looks like I am good at 182mb 🙂

Nick Stares18:02:51

However I'm having trouble understanding step 3, in the babashka holy lambda tutorial, where does it show how to create a layer? I've already run through the tutorial before and deployed an example without pods but I don't understand the layer creation process

Nick Stares18:02:37

Are you saying to add another Resource in template.yml like so?

    Type: AWS::Serverless::LayerVersion
      LayerName: BabashkaDepsLayer
      ContentUri: ./.holy-lambda/bb-clj-deps

    Type: AWS::Serverless::LayerVersion
      LayerName: BabashkaPodsLayer
      ContentUri: ./.holy-lambda/pods

Nick Stares18:02:41

and for step 4 do you mean this?

    Type: AWS::Serverless::Function
      FunctionName: AmcUtilityFunction
      Handler: com.revivemedia.amc-utility.core.ExecuteWorkflow
      # For docker based deployments use:
      # PackageType: Image
      # and remove Runtime parameter
      CodeUri: src
          Type: HttpApi
            ApiId: !Ref ServerlessHttpApi
            Path: /
            Method: GET
        - arn:aws:lambda:us-east-1:430000000:layer:holy-lambda-babashka-runtime-amd64:3
        - !Ref BabashkaPodsLayer
        - !Ref BabashkaDepsLayer

Nick Stares18:02:02

sam local invoke builds the layers but I get

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Could not find namespace:
Location: 19:1
I verified that the pods are in .holy-lambda/pods

Karol Wójcik19:02:08

How do you require the pod?

Karol Wójcik19:02:15

This is the issue of how you do require the pod.

Nick Stares19:02:45

(require '[ :as aws]
         '[ :as credentials]
         '[pod.babashka.buddy.codecs :as codecs]
         '[pod.babashka.buddy.hash :as hash]
         '[pod.babashka.buddy.mac :as mac]
         '[pod.babashka.postgresql :as pg])

Karol Wójcik19:02:56

Have you loaded the pod before the require?

Nick Stares19:02:22

Ah! I had that before but I took it out because I thought that was happening in the sync process. Added it back in and it's working now. Thank you so much!

Nick Stares19:02:53

If you think it would be helpful I will consider making a PR to the docs spelling this out so it is more obvious for someone with my experience level with Lambda layers

Karol Wójcik19:02:58

Yes please. I really need PRs for documentation. If you willing to make the docs better please do. I’m migrating the docs to docosaurus since the formatting of the current solution is far from acceptable. I’m willing to create better, more complete docs, but still help is much appreciated.

👍 1
Nick Stares01:02:37

Strange, apparently I am exceeding the dependency size Resource handler returned message: "Layers consume more than the available size of 262144000 bytes Even though bb-clj-deps and pods are smaller than 250mb. Is it also including .babashka somehow?

Nick Stares01:02:11


[email protected] .holy-lambda % du -hs $(ls -A)
 12K	.DS_Store
179M	.babashka
8.3M	bb-clj-deps
174M	pods

Nick Stares02:02:52

If I needed to, how would I pack the lambda as a docker image?

Karol Wójcik05:02:00

There is also a layer that includes babashka as a binary.

Karol Wójcik05:02:51

Wouldn't you want to try native runtime instead?

Karol Wójcik05:02:59

You got three options: 1. Remove one pod from the layer, so it will get redownloaded during the execution. 2. Create a docker image, but this is not documented and I will have to guide you how to do this. 3. You can use native runtime instead

Karol Wójcik07:02:26

> If I needed to, how would I pack the lambda as a docker image? 1. Create a Dockerfile 2. Set a workdir to /opt 3. Take a zip of babashka layer and copy the content to /opt directory 4. Change workdir to /var/task and copy all the content of dependencies and pods to this directory 5. After this you would have to plug the Dockerfile in template.yml config 6. Try to build the image via sam build 7. Deploy the function via sam deploy -g to ensure a image registry creation.

Karol Wójcik13:02:59

Btw it might be easier to use native backend instead

Nick Stares17:02:20

Ok maybe I will give native backend a shot

Karol Wójcik19:02:38

Cool I’m happy to help ;)

Karol Wójcik19:02:12

The first time seems hard, but when you get used to the process you will be quite happy with the results ;)

Karol Wójcik19:02:20

Which service are you planning to use with AWS cognitect api?

Nick Stares19:02:33

Thanks so much for offering to help! I think the first step is to build one of the native examples, maybe the sqs one

Nick Stares19:02:33

Right now I'm using STS (maybe not necessary since I may be able to get the session tokens I need from KMS within lambda?), S3 and SQS

Karol Wójcik19:02:19

Fine. You therefore need to have 3 clients and at least one stub invoke of each client to correctly gather the resources of AWS Cognitect API during bb hl:native:conf call

👍 1
Nick Stares20:02:43

not sure why this is happening:

[email protected] sqs.example % bb hl:doctor
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: hl:doctor

Nick Stares20:02:54

This appears to work but then gives the same error:

nic[email protected] sqs.example % bb hl:compile
Checking out:  at 2098fc6860948bc854a4f9bc55a0dc6f45ea4901
Downloading: org/clojure/clojure/1.10.3/clojure-1.10.3.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.pom from central
Downloading: org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Downloading: org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.pom from central
Downloading: org/clojure/clojure/1.10.3/clojure-1.10.3.jar from central
Downloading: org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.jar from central
Downloading: org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.jar from central
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: hl:compile

Karol Wójcik20:02:34

Examples might not be up to date. Create a new project using template

Karol Wójcik20:02:51

Then copy paste sources and deps

Karol Wójcik20:02:57

Essentially the errors you are getting comes from bb tasks.

Karol Wójcik20:02:13

Yeah. This example comes from HL < 0.6.0

Karol Wójcik20:02:23

I should definetely update those.

Nick Stares20:02:12

Excuse my ignorance since I've pretty much only used lein , running this gives

[email protected] code % clojure -X:new :template holy-lambda :name :output holy-lambda-example
No function found on command line or in :exec-fn

Karol Wójcik20:02:24

I’ll update the example and set up a minimal project with sqs, sts, s3, kms for you before work, but I’ll be able to do it only in 8-9 hours :)

❤️ 1
Nick Stares20:02:44

Wow, that is so kind of you! Thank you

Karol Wójcik20:02:02

You can use lein new holy-lambda some-project if you like lein more

👍 1
Nick Stares20:02:41

Cool, I am open minded to using clj , just lacking experience

Karol Wójcik20:02:23

The error you are getting means: „there is no „new” command in your user profile in global home deps.edn”.

Nick Stares20:02:58

Ah, I'll go set that up

Karol Wójcik20:02:54

You can use a T alias if you have the lastest Clj CLI

👍 1
Karol Wójcik20:02:35

I’m going to sleep. Will ping you tomorrow.

❤️ 1
Nick Stares20:02:15

Thanks again so much for your help. Sleep well

Nick Stares01:02:28

So I was able to successfully deploy with the clojure backend! 🙂 Still interested in switching to native for perf but this is a good start. However for whatever reason I'm not able to access the environment variables that I need in prod

{:AccessKeyId     (System/getenv "AWS_ACCESS_KEY_ID")
   :SecretAccessKey (System/getenv "AWS_SECRET_ACCESS_KEY")
   :SessionToken    (System/getenv "AWS_SESSION_TOKEN")})
Apparently it's not possible to get them with sts but they are supposed to be in these env variables, not sure what the issue is.

Nick Stares01:02:20

I wanted to test it with sam local invoke as well but I wasn't sure how to get those env vars in that environment either

Nick Stares01:02:28

Btw the reason I need these is that I am using an AWS private endpoint and I am just writing an http request myself because I couldn't figure out how to do it with aws-cli

Nick Stares01:02:57

sqs is working as is, but I can't make the http request I need to push the correct data onto the sqs message body without those credentials

Karol Wójcik06:02:28

Here is an updated version: 1. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY are always present in Lambda production environment. To ensure Lambda may call a service X you have to add set of permissions in template.yml. This way AWS Lambda Service will be able to call service X. Lambda Service automatically creates a role for Lambda with the given permissions and populates the mentioned variables in function execution context. 2. When it comes to local environment your Lambda has the same permissions as your user specified in default profile. You can emulate a different used by switching a --profile in sam local invoke command. I don't understand your use case nor what is not working for you 😄 > sqs is working as is, but I can't make the http request I need to push the correct data onto the sqs message body without those credentials Could you elaborate?

❤️ 1
Karol Wójcik06:02:06

What is behind this AWS private endpoint?

Nick Stares18:02:32

Hey just saw this. Not sure why notifications weren't working on my phone. I was OOO for the holiday weekend in USA. Thanks so much for building out that example! I can see that the env vars are there so let me dig into the permissions and see if I set them up properly

Nick Stares18:02:14

Behind the AWS private endpoint is a service called Amazon Marketing Cloud which is in private beta at the moment.

Nick Stares18:02:24

The problem is that babashka.curl/post is returning nil and I can't figure out why, maybe it's not an authorization issue but I don't know how I would determine that

Karol Wójcik19:02:56

Is there a chance you can determine if actual network call is made?

Nick Stares19:02:06

How could I see if a network call is made?

Karol Wójcik19:02:25

Have you checked if curl could be replace with this

Nick Stares19:02:28

Probably could, let me give it a shot

Karol Wójcik19:02:45

First thing I would try is to make a HTTP call locally

Karol Wójcik19:02:55

To this endpoint with the fake data

Karol Wójcik19:02:09

I mean I expect it's not a fault of Lambda or HL. It's either issue with your code or in curl, so lets narrow the scope of the test.

Nick Stares19:02:48

Oh yeah, I'm able to make the call locally, just not working in production

Nick Stares19:02:03

Ah, I logged the credentials (probably bad practice) and then plugged them into my local setup and I got

"{\"Message\":\"User: arn:aws:sts::***:assumed-role/sam-app-AmcUtilityFunctionCljRole-1J5RNRMA9YX2O/AmcUtilityFunctionClj is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:********3042:ry8g3pqgz4/prod/POST/workflowExecutions\"}",
So looks like I need to set that authorization :thumbsup:

Nick Stares19:02:40

Annoying that I couldn't just get it to log that message in prod though. Maybe switching to clj-http-lite will make that easier

Karol Wójcik20:02:07

Hard to tell. Maybe there is some kind of regression in Babashka curl in Amazon Linux. As far as I remember the Babashka curl is writing the headers file to temp. I dont remember whether it is or is not supported in Lambda environment. If I were you I would make a test with basic GET HTTP call to some known domain like Google with additional header to make sure this is not an internal issue in bb curl.

Nick Stares01:02:33

Wow, the problem was very silly on my end. I had a function called request which was being shadowed by [{:keys [event ctx] :as request}] in the scope of the lambda function

Nick Stares23:02:41

For a second lambda fn, do you recommend another file in the same project or a new project? How does that work with the docker image build process?

Karol Wójcik06:02:27

1. Question: are you using Babashka runtime in docker? 2. How you do set a handler for a lambda function?

Karol Wójcik06:02:36

Anyway, if your functionality is similiar then just put another defn in your code and reference this defn in entrypoint macro. As a next step point to this new handler in deployment descriptor of a new Lambda. Basically you can override the CMD (that is in Dockerfile) in template.yml or AWS Console. No need to create a separate project.

👍 1
Nick Stares17:02:37

Ah, I switched to clojure backend.

Nick Stares19:03:59

Heya, this isn't a HL specific question but I wonder if you might have any advice about it. I am trying to deploy my lambdas to two separate stacks for testing and production. However when I run sam deploy --stack-name amc-test , it fails because the Lambas already exist by that name in my existing stack. How do you typically approach this problem?

Karol Wójcik19:03:27

Reference the stack name in a function name via !Sub.

Nick Stares19:03:47

Like in my template.yml when I declare the functions?

Nick Stares01:03:19

Thanks so much 🙂 I got it to work!