Clojurians Log v2

ok I’m back and playing with it again

trying to be methodically, here’s my plan…

## Env

- M1 mac
- docker desktop
- bb 
- aws-vault (not required until deploy. not using HL for deploy)

## Goals

1. sample using :babashka running locally (invoke and api)
2. sample using :native compile locally
3. sample using :native running locally (invoke and api)
4. sample using :native deployed to AWS

5. mylambda running locally (api) - discover best runtime for REPL connection / interactive changes?
6. mylambda compiled locally
7. mylambda deployed to AWS
   
8. mylambda compiled in CI deployed to AWS. Winning!

does any of this look difficult to you?

I suspect using my M1 mac is causing problems….

➜  holy-lambda-example bb stack:sync
[holy-lambda] Command <stack:sync>
[holy-lambda] Syncing project and holy-lambda deps.edn
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[holy-lambda] Babashka pods found! Syncing babashka pods. Pods should be distributed via a layer which points to .holy-lambda/pods
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Could not find /project/.holy-lambda/clojure-tools-1.10.3.849.jar
Attempting download from 
Cloning: 

this hangs at the last step. in Docker desktop I see a warning about the image used in the container “image may have poor performance or fail if running via emulation”. the process in the container pegs 2 cores at 100% and never completes so I have to kill it

I wonder if this will only work on an Intel Mac?

before all of that: when is the best time to reach out to you for help? I’m aware that you probably have a job 🙂

@steveb8n you can write to me anytime, however if you would like to setup a call, then 4pm Warsaw time is most optimal for me. I will answer all the questions starting from the bottom: 1. warning you see should not stop you from using HL as far as I can tell. You’re probably the only user of HL with M1, but still it should not be a problem (at least I hope so). 2. The last step is cloning in which it hangs? How long did you wait? What I can see is that you’re trying babashka runtime which requires a special layer to work. During stack:sync HL tries to deploy the layer for you, but it’s not possibile since you have fake credentials. My sugestiom are: • set :self-manage-layers? option in bb.edn to true • manually deploy the layer: https://serverlessrepo.aws.amazon.com/#!/applications/eu-central-1/443526418261/holy-lambda-babashka-runtime and reference it’s ARN. • Remove all the :pods for now, since you probably not using any Regarding invoke/api do you mean bb stack:invoke and bb stack:api?

ok. you are right that I don’t really want layers. I’m just using :babashka to prove the minimal case.

what I really want is :native i.e. no layers

Regarding goals, they seem rather typical I would say. I would probably have some hints regarding 5) which we can discuss further on call.

should I skip the minimal :babashka goals in that case?

Yep

I would do that

I would try :java runtime at first

ok. I’ll start with :native right away

4pm your time is midnight my time. the earth is working against our collaboration

If I get really stuck then I’ll gladly stay up late for it but, for now, I’ll try iterating via slack like this

I’ll try :java now

Tomorrow I am available the whole day, so we can arrange a call anytime you want

Btw how do you test your lambas locally with cdk?

my current lambdas are cljs. I run them inside a node local server and I wrote a layer to mock api-gw requests into thek

them

I like the idea of using SAM instead

You would have to make sure that lambdas are declared in template.yml if you would like to use bb stack:api, bb stack:invoke

my deeper goal is to fix slow cold-starts because I have a chain of 3 lambdas in my prod system. an authorizer, a dispatcher and a runner. 3 cljs cold starts is way too long so I’m gonna replace 2 of the 3 (not the runner) with a native version

hmm, looks like a similar hang for stack:sync using :java. I’ll wait a while and see if it gets any further

I waited > 10 mins last time

re testing: I’ll do most of it using mocked requests and aws api call with .cljc sources/tests

but having a local runtime is also needed for running the full stack on a dev machine

I’m worried about hanging stack:sync.

Please try bb stack:prune, bb stack:sync. Show me please the output of bb stack:doctor.

me too. it possible that it’s a docker problem. the docker desktop for M1 is still beta

it completed once when I first tried it. I might restart the docker daemon and see if that helps

same hang now. > 5 mins in. no network traffic, 2 cores at 100%. I’ll kill it and try a restart

Do you use VPN?

Try to but :network ”host” in your :docker option.

It seems like it hangs on downloading the dependencies.

no VPN

Use bb stack:prune, show me the output of doctor and run sync again.

Btw what memory limit did you specify in GUI?

my ISP had an outage this afternoon, so I could have a messed up DNS cache on this machine. I think I’ll try a full reboot to rule that out

using “host” didn’t change behaviour

The clojure -P hangs there

This is what I’m using in docker

Try to run it locally

➜  holy-lambda-example bb stack:prune
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: stack:prune

It should be stack:purge then.

➜  holy-lambda-example bb stack:doctor
[holy-lambda] Command <stack:doctor>

[holy-lambda] ---------------------------------------
[holy-lambda]  Checking health of holy-lambda stack
[holy-lambda]  Home directory is:        /Users/steve
[holy-lambda]  Project directory is:     /Users/steve/Documents/oss/holy-lambda-example
[holy-lambda]  AWS SAM version:          SAM CLI, version 1.26.0
[holy-lambda]  AWS CLI version:          aws-cli/2.2.17 Python/3.9.6 Darwin/20.4.0 source/arm64 prompt/off
[holy-lambda]  AWS directory is:         /Users/steve/.aws
[holy-lambda]  AWS directory exists?:    true
[holy-lambda]  Docker version:           Docker version 20.10.7, build f0df350
[holy-lambda]  Babashka tasks sha:       f0fb9a181445a8b153f11895475ad76cf8a5da5e
[holy-lambda]  Babashka tasks version:   0.2.2
[holy-lambda]  Babashka version:         babashka v0.4.6
[holy-lambda]  Runtime:                  :java
[holy-lambda]  Runtime version:          0.1.2
[holy-lambda]  TTY:                      true
[holy-lambda]  Runtime entrypoint:       com.company.example-lambda.core
[holy-lambda]  Stack name:               example-lambda-52640d-stack
[holy-lambda]  S3 Bucket name:           example-lambda-52640d
[holy-lambda]  S3 Bucket prefix:         holy-lambda
[holy-lambda]  S3 Bucket exists?:        false
[holy-lambda] ---------------------------------------

[holy-lambda] :runtime looks good
[holy-lambda] :runtime:entrypoint looks good
[holy-lambda] :stack:capabilities looks good
[holy-lambda] :runtime:bootstrap-file is supported only for :native runtime
[holy-lambda] :runtime:native-deps is supported only for :native runtime
[holy-lambda] :runtime:version is supported only for :babashka runtime
[holy-lambda] :runtime:pods are supported only for :babashka runtime
[holy-lambda] :runtime:native-image-args are supported only for :native runtime
[holy-lambda] :stack:name looks good
[holy-lambda] property :mvn/local-repo in file deps.edn is correct
[holy-lambda] property :mvn/local-repo in file bb.edn is correct
[holy-lambda] Syncing stack is not required
[holy-lambda] :infra:region definition looks good
[holy-lambda] :infra:bucket-prefix looks good
[holy-lambda] :stack:template looks good
[holy-lambda] :infra:bucket-name looks good, but example-lambda-52640d does not exists (use bb :bucket:create)
[holy-lambda] Required commands ["aws" "sam" "bb" "docker" "clojure" "zip" "id" "clj-kondo" "bash"] installed!

[holy-lambda] Validating template.yml
2021-07-07 19:14:57 Loading policies from IAM...
Traceback (most recent call last):
  File "/opt/homebrew/bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 153, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 122, in wrapped
    return_value = func(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/lib/utils/version_checker.py", line 42, in wrapped
    actual_result = func(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/cli/main.py", line 90, in wrapper
    return func(*args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/commands/validate/validate.py", line 37, in cli
    do_cli(ctx, template_file)  # pragma: no cover
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/commands/validate/validate.py", line 59, in do_cli
    validator.is_valid()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samcli/commands/validate/lib/sam_template_validator.py", line 56, in is_valid
    managed_policy_map = self.managed_policy_loader.load()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/samtranslator/translator/managed_policy_translator.py", line 25, in load
    for page in page_iterator:
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.26.0/libexec/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the ListPolicies operation: The security token included in the request is invalid.

does this mean that it’s still trying to reach aws?

purge was much better

same result from doctor. failing when trying to get IAM policies

would that explain the hang?

Nope

I won’t bother rebooting as this seems like the culprit right?

I’m trying to validate the template via sam which requires aws credentials. Ignore it

It’s not connected at all

Can you use clojure -P? Does it hang?

yes clojure -P runs ok. not sure what to expect from it

stack sync is now moving (after purge) i.e. downloading deps

and sync complete!

Ok good.

next try stack:invoke?

Yep

Or bb stack:api ;)

Whatever you would like.

missing a jar. I suspect I need a build first

Yep

It’s good that tasks are informative

running stack:compile now

Great!

Btw i will add an entry to troubleshooting guide. It seems that docker process somewhat hanged last time.

4 cores working very hard

➜  holy-lambda-example bb stack:compile
[holy-lambda] Command <stack:compile>
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[main] INFO hf.depstar.uberjar - Compiling com.company.example-lambda.core ...
[main] INFO hf.depstar.uberjar - Building uber jar: .holy-lambda/build/output.jar
➜  holy-lambda-example 

yeah the purge trick is good for troubleshooting

It hangs on compile?

no compile completed fine

[holy-lambda] Command <stack:invoke>ke 
[holy-lambda] ----------------------------- Runtime Output: ----------------------------
[holy-lambda] Invoking com.company.example-lambda.core.ExampleLambda (java8)
Decompressing /Users/steve/Documents/oss/holy-lambda-example/.holy-lambda/build/output.jar
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-java8:rapid-1.26.0.

Mounting /private/var/folders/x3/v5swftpj3gx3bt3wvlzkzwy80000gn/T/tmpkofppcl6 as /var/task:ro,delegated inside runtime container
START RequestId: ed161c24-e703-4a62-83b9-543d6e3b412b Version: $LATEST
Function 'ExampleLambdaFunction' timed out after 40 seconds
No response from invoke container for ExampleLambdaFunction

[holy-lambda] -------------------------------------------------------------------------

[holy-lambda] ----------------------------- Function Output: --------------------------
[holy-lambda] 
[holy-lambda] -------------------------------------------------------------------------
➜  holy-lambda-example 

I’m out of my depth trying to understand that last log

Function times out

is it that the jvm took too long to start?

How much memory did you set on docker?

I wonder if M1 docker emulation is just really slow?

It should be at least 8 GB

ok. did I not RTFM?

That’s first thing

kinda :D

sorry 🙂

Did you change anything in the implementation of lambda?

no changes to code from clj-new

Use 80 seconds instead of 40 in template.yml.

ok. trying 80

Btw it looks like docker support on M1 is quite bad now

I have an intel OSX laptop also. I can try this there

it’s just half as fast

80 secs worked!

[holy-lambda] ----------------------------- Function Output: --------------------------
[holy-lambda] {"statusCode":200,"headers":{"content-type":"text/plain; charset=utf-8","Custom-Header":"Some Value"},"body":"Hello world"}
[holy-lambda] -------------------------------------------------------------------------

XD

hi five

That’s make me laugh

trying api now

👋🙌

Remember that you have to visit the site to invoke lambda

the irony of doing this to get faster cold starts on aws

On my machine :java runtime takes like 9s to respond. This is the response time I see on AWS as well.

api working now too

once loaded, api response is instant

I can check off a couple of goals now

thanks. I gotta go eat my dinner now too

You can use native:executable to produce native artifact

ok, trying that before dinner

Make sure to use it with :runtime :native

the cli told me to do that. good error handling

Like: bb native:executable :runtime :native

waiting on big graal command now…

maybe no surprise that graal is slow in a slow docker container

Let me know when it finishes. You can add some cores to docker as well. Maybe it could help a least a little bit.

yeah this is gonna take a while. still going strong

I can always try the native build on the intel laptop. just do local dev on the M1

I see what I can learn about the docker emulation/perf on M1 as well. maybe something I can do about that

but, for now I have enough to progress so I’ll carry on with it tomorrow first thing

I’ll touch base with you again tomorrow to discuss getting a good combo of SAM/REPL/local going

thanks for the help! I will pay this forward with my own OSS project soon

Sure! We can settle a call for this! ;)

great. have a good day

U2

do I have no choice but to create a default profile? I don’t want this because it’s much less secure than using aws-vault

I looked up the check done for aws access and tried that…

➜ holy-lambda-example aws configure get region --profile default ap-southeast-2 ➜ holy-lambda-example echo $? 0 ➜ holy-lambda-example

zero exit code in zsh but for some reason the bb task still rejects it

my goal is to run local (native) and compile. I don’t need aws access for this right?

You don’t need AWS access for this

great. that’s what I thought

but bb tasks returns….

➜ holy-lambda-example bb tasks The following tasks are available: [holy-lambda] AWS configuration check failed. Unable to get value from the profile: default Did you run command: aws configure? ➜ holy-lambda-example

What bb task are you trying?

no task yet. just “bb tasks”

Hmm.. i will put this check in bb stack:doctor. I did not realize that some people don’t use profiles

How AWS vault works with AWS sam?

not sure, I haven’t used SAM before. just installed it now

but I use CDK for deploys so no SAM until HL

Ok got it

Give me 2 minutes please

no probs

in fact, I’m going out for an hour. will check back later

Question: cannot you just put anything in profile?

not gone yet 🙂 not sure but I use ~/.aws/config and yes, I can put anything I want in there

.aws/crendentials

[default]
aws_access_key_id = hello
aws_secret_access_key = world

Put it there

It should work that way I think

https://github.com/99designs/aws-vault#roles-and-mfa

ok I’ll see if that works and doesn’t break aws-vault

Just checked AWS vault

It seems that I should check whether environment variables are set

cool. bb tasks now works

stack:sync is running ok too

Cool. You can progress now 🙂

it did warn me that I’m on an M1 mac but I’ll play with it now and see what I can learn

What did warn you?

It's not my warning 😄

I think it’s from docker but I’ll keep digging

aws-vault not broken with your workaround so that’s good too

ok. I’m heading out now. will let you know how I get on later

As long as you don't touch any tasks like bb bucket:create, bb stack:deploy, bb stack:destroy, bb bucket:remove, bb bucket:pack it will work 🙂

In meantime I will work to take secret, access keys from environment variables if possible