Clojurians Log v2

Heya, this isn't a HL specific question but I wonder if you might have any advice about it. I am trying to deploy my lambdas to two separate stacks for testing and production. However when I run sam deploy --stack-name amc-test , it fails because the Lambas already exist by that name in my existing stack. How do you typically approach this problem?

Reference the stack name in a function name via !Sub.

Like in my template.yml when I declare the functions?

Yeah.

Thanks so much 🙂 I got it to work!

Ah, I switched to clojure backend.

Cc @nickstares0

Do you mean a layer of dependencies?

Exactly

1. Put the symbol/version pairs to bb.edn configuration in :pods keyword. 2. Run bb hl:babashka:sync to download the pods to a directory 3. Make a layer the same way as it's specified in documentation (babashka holy lambda tutorial), but point to .holy-lambda/pods 4. Link the layer with your function by a reference. 5. Verify if it works by using sam local invoke.

holy-lambda has it's own mechanism of downloading the pods, so the pods will never get redownloaded in Lambda environment.

Also check if your dependencies are not bigger than 250mb, since otherwise you would have to pack the lambda in docker image.

Looks like I am good at 182mb 🙂

However I'm having trouble understanding step 3, in the babashka holy lambda tutorial, where does it show how to create a layer? I've already run through the tutorial before and deployed an example without pods but I don't understand the layer creation process

Are you saying to add another Resource in template.yml like so?

Resources:
  BabashkaDepsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: BabashkaDepsLayer
      ContentUri: ./.holy-lambda/bb-clj-deps

  BabashkaPodsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: BabashkaPodsLayer
      ContentUri: ./.holy-lambda/pods

and for step 4 do you mean this?

AmcUtilityFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: AmcUtilityFunction
      Handler: com.revivemedia.amc-utility.core.ExecuteWorkflow
      # For docker based deployments use:
      # PackageType: Image
      # and remove Runtime parameter
      CodeUri: src
      Events:
        HelloEvent:
          Type: HttpApi
          Properties:
            ApiId: !Ref ServerlessHttpApi
            Path: /
            Method: GET
      Layers:
        - arn:aws:lambda:us-east-1:430000000:layer:holy-lambda-babashka-runtime-amd64:3
        - !Ref BabashkaPodsLayer
        - !Ref BabashkaDepsLayer

sam local invoke builds the layers but I get

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Could not find namespace: pod.babashka.aws.
Location: 19:1

Looks good.

How do you require the pod?

This is the issue of how you do require the pod.

(require '[pod.babashka.aws :as aws]
         '[pod.babashka.aws.credentials :as credentials]
         '[pod.babashka.buddy.codecs :as codecs]
         '[pod.babashka.buddy.hash :as hash]
         '[pod.babashka.buddy.mac :as mac]
         '[pod.babashka.postgresql :as pg])

Have you loaded the pod before the require?

Ah! I had that before but I took it out because I thought that was happening in the sync process. Added it back in and it's working now. Thank you so much!

If you think it would be helpful I will consider making a PR to the docs spelling this out so it is more obvious for someone with my experience level with Lambda layers

Yes please. I really need PRs for documentation. If you willing to make the docs better please do. I’m migrating the docs to docosaurus since the formatting of the current solution is far from acceptable. I’m willing to create better, more complete docs, but still help is much appreciated.

Strange, apparently I am exceeding the dependency size Resource handler returned message: "Layers consume more than the available size of 262144000 bytes Even though bb-clj-deps and pods are smaller than 250mb. Is it also including .babashka somehow?

better:

nick@Nicks-MacBook-Pro-2 .holy-lambda % du -hs $(ls -A)
 12K	.DS_Store
179M	.babashka
8.3M	bb-clj-deps
174M	pods

If I needed to, how would I pack the lambda as a docker image?

There is also a layer that includes babashka as a binary.

Wouldn't you want to try native runtime instead?

You got three options: 1. Remove one pod from the layer, so it will get redownloaded during the execution. 2. Create a docker image, but this is not documented and I will have to guide you how to do this. 3. You can use native runtime instead

> If I needed to, how would I pack the lambda as a docker image? 1. Create a Dockerfile 2. Set a workdir to /opt 3. Take a zip of babashka layer and copy the content to /opt directory 4. Change workdir to /var/task and copy all the content of dependencies and pods to this directory 5. After this you would have to plug the Dockerfile in template.yml config 6. Try to build the image via sam build 7. Deploy the function via sam deploy -g to ensure a image registry creation.

Hey just saw this. Not sure why notifications weren't working on my phone. I was OOO for the holiday weekend in USA. Thanks so much for building out that example! I can see that the env vars are there so let me dig into the permissions and see if I set them up properly

Behind the AWS private endpoint is a service called Amazon Marketing Cloud which is in private beta at the moment.

The problem is that babashka.curl/post is returning nil and I can't figure out why, maybe it's not an authorization issue but I don't know how I would determine that

Is there a chance you can determine if actual network call is made?

How could I see if a network call is made?

Have you checked if curl could be replace with this https://github.com/clj-commons/clj-http-lite/tree/master

Probably could, let me give it a shot

First thing I would try is to make a HTTP call locally

To this endpoint with the fake data

I mean I expect it's not a fault of Lambda or HL. It's either issue with your code or in curl, so lets narrow the scope of the test.

Oh yeah, I'm able to make the call locally, just not working in production

Ah, I logged the credentials (probably bad practice) and then plugged them into my local setup and I got

"{\"Message\":\"User: arn:aws:sts::***:assumed-role/sam-app-AmcUtilityFunctionCljRole-1J5RNRMA9YX2O/AmcUtilityFunctionClj is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:********3042:ry8g3pqgz4/prod/POST/workflowExecutions\"}",

Annoying that I couldn't just get it to log that message in prod though. Maybe switching to clj-http-lite will make that easier

Hard to tell. Maybe there is some kind of regression in Babashka curl in Amazon Linux. As far as I remember the Babashka curl is writing the headers file to temp. I dont remember whether it is or is not supported in Lambda environment. If I were you I would make a test with basic GET HTTP call to some known domain like Google with additional header to make sure this is not an internal issue in bb curl.

Wow, the problem was very silly on my end. I had a function called request which was being shadowed by [{:keys [event ctx] :as request}] in the scope of the lambda function

Cool 🙂

For a second lambda fn, do you recommend another file in the same project or a new project? How does that work with the docker image build process?

1. Question: are you using Babashka runtime in docker? 2. How you do set a handler for a lambda function?

Anyway, if your functionality is similiar then just put another defn in your code and reference this defn in entrypoint macro. As a next step point to this new handler in deployment descriptor of a new Lambda. Basically you can override the CMD (that is in Dockerfile) in template.yml or AWS Console. No need to create a separate project.

Btw it might be easier to use native backend instead

Ok maybe I will give native backend a shot

Cool I’m happy to help ;)

The first time seems hard, but when you get used to the process you will be quite happy with the results ;)

Which service are you planning to use with AWS cognitect api?

Thanks so much for offering to help! I think the first step is to build one of the native examples, maybe the sqs one

Right now I'm using STS (maybe not necessary since I may be able to get the session tokens I need from KMS within lambda?), S3 and SQS

Fine. You therefore need to have 3 clients and at least one stub invoke of each client to correctly gather the resources of AWS Cognitect API during bb hl:native:conf call

not sure why this is happening:

nick@Nicks-MacBook-Pro-2 sqs.example % bb hl:doctor
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: hl:doctor

This appears to work but then gives the same error:

nick@Nicks-MacBook-Pro-2 sqs.example % bb hl:compile
Checking out:  at 2098fc6860948bc854a4f9bc55a0dc6f45ea4901
Downloading: org/clojure/clojure/1.10.3/clojure-1.10.3.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.pom from central
Downloading: org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Downloading: org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.pom from central
Downloading: org/clojure/clojure/1.10.3/clojure-1.10.3.jar from central
Downloading: org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.jar from central
Downloading: org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.jar from central
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: hl:compile

Examples might not be up to date. Create a new project using template

Then copy paste sources and deps

Essentially the errors you are getting comes from bb tasks.

Yeah. This example comes from HL < 0.6.0

I should definetely update those.

Gotcha

Excuse my ignorance since I've pretty much only used lein , running this gives

nick@Nicks-MBP-2 code % clojure -X:new :template holy-lambda :name com.company/example-lambda :output holy-lambda-example
No function found on command line or in :exec-fn

I’ll update the example and set up a minimal project with sqs, sts, s3, kms for you before work, but I’ll be able to do it only in 8-9 hours :)

Wow, that is so kind of you! Thank you

You can use lein new holy-lambda some-project if you like lein more

Cool, I am open minded to using clj , just lacking experience

The error you are getting means: „there is no „new” command in your user profile in global home deps.edn”.

Ah, I'll go set that up

https://github.com/seancorfield/clj-new

You can use a T alias if you have the lastest Clj CLI

I’m going to sleep. Will ping you tomorrow.

Thanks again so much for your help. Sleep well

So I was able to successfully deploy with the clojure backend! 🙂 Still interested in switching to native for perf but this is a good start. However for whatever reason I'm not able to access the environment variables that I need in prod

{:AccessKeyId     (System/getenv "AWS_ACCESS_KEY_ID")
   :SecretAccessKey (System/getenv "AWS_SECRET_ACCESS_KEY")
   :SessionToken    (System/getenv "AWS_SESSION_TOKEN")})

I wanted to test it with sam local invoke as well but I wasn't sure how to get those env vars in that environment either

Btw the reason I need these is that I am using an AWS private endpoint and I am just writing an http request myself because I couldn't figure out how to do it with aws-cli

sqs is working as is, but I can't make the http request I need to push the correct data onto the sqs message body without those credentials

Here is an updated version: https://github.com/FieryCod/holy-lambda/tree/master/examples/bb/native/cognitect.aws.api/src/sqs/example 1. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY are always present in Lambda production environment. To ensure Lambda may call a service X you have to add set of permissions in template.yml. This way AWS Lambda Service will be able to call service X. Lambda Service automatically creates a role for Lambda with the given permissions and populates the mentioned variables in function execution context. 2. When it comes to local environment your Lambda has the same permissions as your user specified in default profile. You can emulate a different used by switching a --profile in sam local invoke command. I don't understand your use case nor what is not working for you 😄 > sqs is working as is, but I can't make the http request I need to push the correct data onto the sqs message body without those credentials Could you elaborate?

What is behind this AWS private endpoint?