Fork me on GitHub
#babashka
<
2022-04-11
>
Erkan09:04:40

Hello lovely people! what is the recommended way to bake in pods in a container when using babashka/babashka:latest Docker? Would it be to copy my local ~/.babashka/pods/ into the container?

borkdude09:04:30

@hallstrom.eric Probably build a custom docker image and during build time download the pods into the image

borkdude09:04:02

You can download the pods by just evaluating an expression and having them in your bb.edn

Erkan09:04:25

Yes that I have, it's very nice!

Erkan09:04:38

Is there some flag or something for bb where you can specify where the pods folder is located?

borkdude09:04:55

Not yet. Where would you use this?

Erkan09:04:39

I was thinking if I copy the pod into the container and the specify the location for that pod

Erkan09:04:54

Might as well just copy it into ~/.babashka/pods/ I guess

borkdude09:04:27

You can also make another bb.edn where you specify :path in the pods config

borkdude09:04:53

{:pods {foo/bar {:path "/usr/local/bin/pod"}}}

Erkan09:04:02

ohh that is awesome

Erkan09:04:02

But wait, so the pod should be downloaded just when evaluating the expression?

Erkan09:04:29

Because I'm trying to run the aws pod inside a lambda but having this error

xception in thread "main" java.io.FileNotFoundException: /home/sbx_user1051/.babashka/pods/repository/org.babashka/aws/0.1.2/manifest.edn (No such file or directory)
I assumed it didn't manage to download the pod

borkdude09:04:13

if you use :path then it will not download the pod but expect the pod to be there already

Erkan10:04:58

Is it supposed to look like this:

:pods {org.babashka/aws {:version "0.1.2"
                          :path "/var/task/pods/aws"}
if I have the pod under a directory there; with these files:
manifest.edn     metadata.cache   pod-babashka-aws

Erkan10:04:33

yes, works like a charm! 🙂

Erkan11:04:56

Oh but then I can't require that pod in my namespace with just this

(:require
   [pod.babashka.aws :as aws]))
it seems to be still downloading it

Erkan11:04:31

when you call it org.babashka/aws will it default to the pod registry and go and fetch it from there @U04V15CAJ?

borkdude11:04:57

Yes, unless you use :path

borkdude11:04:48

Path should point at the binary

borkdude11:04:55

Not a directory

Erkan11:04:07

So I'm pointing to the binary in bb.edn ->

:pods {org.babashka/aws {:version "0.1.2"
                          :path "./pods/aws/0.1.2/pod-babashka-aws"}}
and in my namespace I require it like this:
(:require [pod.babashka.aws :as aws]))
It still downloads the pod though 😕

borkdude11:04:19

Don't put the version there

borkdude11:04:36

Maybe that helps

Erkan11:04:40

without the version it fails

borkdude11:04:13

It fails how exactly?

Erkan11:04:32

./pods/aws/0.1.2/pod-babashka-aws: 1: : not found
./pods/aws/0.1.2/pod-babashka-aws: 2: Syntax error: word unexpected (expecting ")")

Erkan11:04:21

but with the version in bb.edn it goes to download the pod 😕

Erkan11:04:55

and if I require it inside the namespace like this

(pods/load-pod "./pod-babashka-aws")
I get this error:
` 8: (pods/load-pod "./pod-babashka-aws")
    ^--- Invalid netstring. Unexpected end of input.
` and this is with
babashka v0.8.0
So it seems like it still wants a URL to fetch it from

cap10morgan12:04:10

:version must not be there. It's that or path, not both. So start there. Then you have some other problem. :)

cap10morgan12:04:38

You don't need to explicitly load it with path either, so get rid of that

cap10morgan12:04:44

The next thing to look at is if the binary it's pointed at is a valid executable for that platform

cap10morgan12:04:24

It's throwing some errors trying to execute the pod binary it looks like

Erkan12:04:35

Hmm ok; I took the binary from my local pod that was under : ~/.babashka/pods/repository/org.babashka/aws/0.1.2/ should I create the binary by building the git repo instead?

borkdude12:04:29

What OS and arch are you on and what OS and arch is the lambda on? They should match

Erkan13:04:45

It's running inside a docker with bb latest image

borkdude13:04:04

I mean your local OS / arch and the other OS / arch.

Erkan17:04:19

I'm running macos monterey, the other OS is on aws lambda where I started the thingy in a container like this -

Dockerfile

FROM babashka/babashka:latest

WORKDIR /var/task

copy app/src src
copy app/bb.edn .
copy app/src/lambda/bootstrap .

copy app/pods/aws/0.1.2/pod-babashka-aws .
RUN chmod +x pod-babashka-aws

CMD ["./bootstrap"]

borkdude17:04:50

Copying a macOS binary to linux won't work

Erkan17:04:14

So then it seems that the container running on aws don't have access to ~/.babashka/pods/ or something

borkdude17:04:10

You can just download the linux binaries when creating the image and then copy them to a path

borkdude17:04:38

I mean in a multi-staged step. @U7ERLH6JX can tell you more about this. Else take a look at #holy-lambda which has support for #babashka + pods

Erkan17:04:11

Ahh smart!

Erkan17:04:23

Didn't know about #holy-lambda!

Erkan18:04:35

Using the linux binary of the pod did of course work; super thank you for the help again!

👍 1
cap10morgan15:04:36

Perhaps there's a way to detect this and give a better error message?

cap10morgan16:04:06

not seeing anything super obvious short of reading binary headers or similar

borkdude16:04:33

Maybe the OS should do a better job here ;)

cap10morgan17:04:06

haha, yeah for sure

borkdude12:05:53

@hallstrom.eric I forgot to mention: > Is there some flag or something for bb where you can specify where the pods folder is located? The environment variables XDG_CACHE and XDG_DATA can be set to influence this

borkdude12:05:51

E.g.:

XDG_DATA=pods bb XDG_CACHE=pods
will download the pods to pods/.babashka/pods/repository/....

borkdude12:05:13

I will add BABASHKA_PODS_DIR for the next release which will cover everything related to pod downloading

markbastian15:04:15

Any recommendations for a time library for bb? Something along the lines of tick.

borkdude15:04:59

@markbastian I recommend using java.time directly, but cljc.time is a library that also works

👍 1
Benjamin15:04:16

/var/task/./pod-babashka-aws: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/./pod-babashka-aws)
/var/task/./pod-babashka-aws: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/./pod-babashka-aws)
when I try to load aws pod in docker image. Using babashka/babashka:latest . Tried apt-get -y install libc6 but that is not it

✔️ 1
lispyclouds15:04:36

what is the base image of the docker image youre using?

Benjamin15:04:55

FROM babashka/babashka:latest as BABASHKA


FROM clojure:tools-deps as BUILDER
ah actually now I see I think in the context of it throwing I'm in this tools-deps image w/e

borkdude15:04:05

Are you running on a M1 mac in Docker perhaps?

Benjamin15:04:43

no. Manjaro (arch)

borkdude15:04:39

I don't understand this error message, when using the (newest) aws pod, this is a static binary

Benjamin15:04:18

ah maybe because I build the pod myself and did it wrong?

Benjamin15:04:30

I'll try with the official one

Benjamin15:04:21

ok that one indeed doesn't throw this err

Benjamin15:04:01

I compiled mine using graalvm-ce-java11-22.0.0.2

Benjamin15:04:44

from the aws pod compile script:

BABASHKA_STATIC=${BABASHKA_STATIC:-}
BABASHKA_MUSL=${BABASHKA_MUSL:-}

if [ "$BABASHKA_STATIC" = "true" ]; then
    args+=("--static")
    if [ "$BABASHKA_MUSL" = "true" ]; then
        args+=("--libc=musl"
               # see 
               "-H:CCompilerOption=-Wl,-z,stack-size=2097152")
    else
        # see 
        args+=("-H:+StaticExecutableWithDynamicLibC")
    fi
fi
maybe I need to set this option so it goes into the the else or something? I will try with the static executable arg

lispyclouds15:04:00

maybe it has a dependency on a version of glibc that the clojure base image doesnt have. we prefer static graal binaries in most places

Benjamin16:04:36

should I set "MUSL" to true as well? I think just using BABASHKA_STATIC=true did not do it

borkdude18:04:57

BABASHKA_MUSL=true yes

borkdude18:04:23

why do you need a self-compiled one?

Benjamin18:04:11

because of the session-token in credentials provider 😅 I a use case where setting the properties is not viable because I'd like to use different clients in the same process

borkdude18:04:23

so you are using a custom fork of the cognitect aws lib?

Benjamin18:04:51

I was just trying to fork the pod atm

Benjamin18:04:12

(defn -kludge-full-basic-credentials-provider
  "Given a map with :access-key-id and :secret-access-key,
  returns an implementation of CredentialsProvider which returns
  those credentials on fetch.

  - Ben: Added session token "
  [{:keys [access-key-id secret-access-key session-token]}]
  (assert access-key-id "Missing")
  (assert secret-access-key "Missing")
  (create-provider
   (reify creds/CredentialsProvider
     (fetch [_]
       {:aws/access-key-id access-key-id
        :aws/secret-access-key secret-access-key
        :aws/session-token session-token}))))
added this to credentials.clj| https://github.com/SingularityGroup/pod-babashka-aws/commit/1716fb326a50676e9ab049fe7c9a126c330ead4a

Benjamin18:04:56

but there might be another workaround with profiles or credentials-proccess

Benjamin19:04:13

building my fork worked now 😄 - thanks. Guess for the real fix we wait for the session-token change upstream

👍 1