Fork me on GitHub
#babashka
<
2023-01-05
>
Jakub Holý (HolyJak)08:01:12

Hello! How can I run a babashka script from a bb task? I could of course (shell "bb ") but that breaks if by a chance the user has given the bb executable a different name… . I guess I could simply put the content of the script into a function such as (defn run! ...) and create a task with :requires ([my-script-file :as scr]), (scr/run!) ? Hm, this does not work. I have next to each other bb.edn, the latter having (ns smoke-test) and bb.edn has now :paths ["." ""] but bb complains it cannot find the smoke-test ns….

2
borkdude08:01:32

load-file?

🙏 2
2
Jakub Holý (HolyJak)08:01:01

Another question. I want to use bb tasks for a smoke test. I have one task to start my server and another to run the smoke test. The question is, how do I stop the server after the smoke test finished? (The task that starts the server is run-app (clojure "-X:sql:dev" "development/start") ) I guess I could replace the run-app tasks by invoking clojure directly via shell/sh asynchronously and then killing it when done…

lispyclouds09:01:28

you could define the start server task separately like i do https://github.com/bob-cd/bob/blob/main/apiserver/bb.edn#L9 and then call it at the right place like https://github.com/bob-cd/bob/blob/main/apiserver/bb.edn#L51

Jakub Holý (HolyJak)09:01:51

though I would still need to find out how to kill the running task…

borkdude09:01:23

Shell returns a process which you can kill with process/destroy

Jakub Holý (HolyJak)09:01:18

but that would require that I use the 2nd approach, of starting the server manually via shell/sh from the smoke test, right?

lispyclouds09:01:22

the (clojure ...) task is shelling out to run a JVM anyways. Not sure if it returns a process but doing (shell "clojure ...") would definitely will

lispyclouds09:01:40

checking if the clojure task gives a process back

borkdude09:01:07

clojure also returns a process that can be destroy-ed

borkdude09:01:05

you might need p/destroy-tree to kill all children processes too

👍 2
Jakub Holý (HolyJak)09:01:59

But can I communicate the output of one task ( the clojure process) to another task, to destroy it there?

borkdude09:01:19

You can have a dependency on that task, then you will have its value

borkdude09:01:10

:depends [run-app]
:task (do (whatever) (p/destroy-tree run-app))

😻 2
borkdude09:01:25

if this sounds too complicated, just write a babashka script to do all of this manually

Jakub Holý (HolyJak)09:01:12

This sounds fine, thank you!

borkdude10:01:12

Yeah, sometimes that's just the easiest way to do things

👍 2
pieterbreed11:01:48

Regarding the grzm/awyeah-api library; does it support aws sso? (configured with aws configure sso) ( @grzm )

pieterbreed11:01:10

I have an environment variable for AWS_PROFILE which works in bash with aws sts get-caller-identity. This env-var is valid inside the bb repl session. I'm getting this output:

DEBUG [com.grzm.awyeah.credentials:?] - Unable to fetch credentials from system properties.
DEBUG [com.grzm.awyeah.credentials:?] - Unable to fetch credentials from aws profiles file.
DEBUG [com.grzm.awyeah.credentials:?] - Unable to fetch credentials from any source.

jeroenvandijk11:01:09

I have a custom (babashka) script that fetches credentials via aws sso. Then elsewhere i insert these credentials in awyeah as session credentials. Is that what you mean?

pieterbreed11:01:32

not really, but I think it would be a work-around maybe... If you set up AWS_PROFILE to a config in your ~/.aws/config which has been configure with aws sso configure, the session is valid, and you are using on of the official AWS SDK's (incl aws cli) ... then you can make API calls. IE - not required to set access-key-id and secret-access-key directly

jeroenvandijk11:01:50

Not sure how aws sso configure works exactly. But im using the official cli tool as well. Via a profile with credential_process it uses my custom script. Which i believe is faster than the official versions

jeroenvandijk11:01:57

Maybe you have a different use case then me. Im mostly using it for scripts that are not running for a very long time

jeroenvandijk11:01:45

Curious to hear what you find though. Ill post what i have as a bbin script later

2
jmglov11:01:05

The way I’ve done this before is using a script that performs the SSO login, then writes the credentials and session token to ~/.aws/config. You can then set AWS_PROFILE to that and awyeah-api’s resolver will work with it. You may need a bit of magic to refresh the credentials once they expire with a long-running REPL session.

pavlosmelissinos11:01:46

This https://github.com/grzm/awyeah-api#differences-from-aws-api looks relevant (a different use case obviously but the same reasoning may apply in your situation): > My use case is short-lived scripts, where JVM start-up time can dwarf script execution time: credentials don't have much time to get stale. If I had longer-lived processes, start-up time wouldn't be an issue and I'd just use the Clojure.

pieterbreed12:01:38

I'm not super worried about refreshing the session once it expires. I've seen the disclaimer and I'm ok with it. For a profile that has been set up with aws configure sso the actual credentials are contained in a file beneath ~/.aws/sso/cache/*.json. so... I think it is imaginable for awyeah to read the credentials from there.

2
👌 2
jmglov12:01:40

This makes sense, but I often have REPL sessions open for days or weeks whilst I’m developing a script, so creds refresh is always something I implement. I’ll look into how to do this with awyeah-api just for fun. 🙂

2
jeroenvandijk12:01:25

@U054AT6KT I have done this in the past with aws-api and a few others too. These two issues give some examples for what I think you want https://github.com/cognitect-labs/aws-api/search?q=credential_process&amp;type=issues

thanks 2
Wanishing12:01:14

You can take a look at https://github.com/wanishing/awyeah-graalvm/blob/main/src/mal/sso_credentials_provider.clj. It’s a minimal example of using s3 client with sso credentials provider.

🆒 2
martinklepsch11:01:38

I’m trying to create a signature with the buddy pod but I’m struggling to replicate this

echo -n test | openssl dgst -sha256 -sign private.pem
with Babashka
(require '[babashka.pods :as pods])

(pods/load-pod 'org.babashka/buddy "0.2.0")

(require '[pod.babashka.buddy.core.codecs :as codecs]
         '[pod.babashka.buddy.core.mac :as mac]
         '[pod.babashka.buddy.core.keys :as keys])

(codecs/bytes->str
  (mac/hash "test"
            {:alg :hmac+sha256
             :key (keys/private-key "private.pem")}))
The resulting value is always much shorter when compared with the value returned by the openssl command

martinklepsch11:01:21

I somehow think I might be doing something wrong with providing the private key but I tried different approaches (`keys/str->private-key`) with similar results

jumar12:01:53

You might be using wrong openssl command? Quickly looking here: https://unix.stackexchange.com/questions/610039/how-to-do-hmacsha256-using-openssl-from-terminal > I got this working. All I had to do is use openssl sha256 instead of openssl dgst -sha256.

martinklepsch12:01:07

hm, the issue is I need what the dgst version gave me

martinklepsch12:01:48

I got that from a reference for an API I’m targeting. So maybe I’m looking at the wrong thing in buddy actually…

jumar12:01:41

Is the output of your function base64 encoded while dgst produces hex-encoded version?

martinklepsch12:01:14

dgst returns bytes which I’m then converting with base64 on the command line

jumar12:01:04

How come? https://www.openssl.org/docs/man1.1.1/man1/openssl-dgst.html > The digest functions output the message digest of a supplied file or files in hexadecimal.

jumar12:01:19

Anyway, you can very quickly see if that's the case from the encoding's alphabet But it could be the reason why your function returns a shorter string.

martinklepsch12:01:23

echo -n test | openssl dgst -sha256 -sign private.pem
��DK��Ɂ���z�VQ[�zxw�-J��Ë�˺�/����V#�� �J���RS�iB;/�j�p۽L..�h˨�մ�d_u�|�̩j�8�\�����;��qT�R����N��7:�L�l$T��[m�~�3����⏎

jumar12:01:36

Ah, I see - I missed the -sign switch. That would be the problem I think. MAC is a symmetric algorithm while you (with the openssl command) seem to be producing an assymetric-cryptography signature

jumar12:01:40

I used something like this before - not sure if that's similar to what you need

(require '[buddy.sign.compact :as compact-sign])
(require '[buddy.core.keys :as buddy-keys])

(def sign-algorithm :es256)
(def signing-keys :start
  {:private-key (buddy-keys/private-key private-key-path)
   :public-key (buddy-keys/public-key public-key-path)})

(compact-sign/sign data (:private-key signing-keys) {:alg sign-algorithm})

martinklepsch12:01:55

ahh that looks very promising!

martinklepsch12:01:31

I guess i was a bit on the wrong track there

martinklepsch16:01:18

Hm, that seems to pull in nippy which makes me think it’s probably not what I’m looking for…

jumar21:01:52

Yes, you are right. After looking into sign implemntation it's obvious that it's doing much more.

(defn sign
  "Sign arbitrary length string/byte array using
  compact sigining method."
  [data key & [{:keys [alg compress]
                :or {alg :hs256 compress true}}]]
  (let [input (serialize data compress)
        salt (nonce/random-bytes 12)
        stamp (codecs/long->bytes (util/now))
        signature (-> (bytes/concat input salt stamp)
                      (calculate-signature key alg))]
    (str/join "." [(bytes->base64str input)
                   (bytes->base64str signature)
                   (bytes->base64str salt)
                   (bytes->base64str stamp)])))
so it not only serializes the data but also adds salt and timestamp However, you can look at calculate-signature and eventually find the signing primitives In my case, I can replace all of it with this:
(def my-private-key (buddy-keys/private-key "my.pem"))
(require '[buddy.core.dsa :as dsa])
(dsa/sign (.getBytes "test") {:alg :ecdsa+sha256 :key my-private-key})
That produces signatures that can be validated with openssl just fine
(let [signature  (dsa/sign (.getBytes "test") {:alg :ecdsa+sha256 :key my-private-key})]
    (with-open [out (io/output-stream (io/file "test.sign.clojure"))]
      (.write out signature)))
in terminal
echo -n test | openssl dgst -sha256 -verify my.pub.pem -signature test.sign.clojure
Verified OK

martinklepsch16:01:41

Just seeing this now.. I ended up with something like this in JVM Clojure:

(defn sign
  "RSA private key signing of a message. Takes message as string"
  [message]
  (let [msg-data (.getBytes message)]
    (->> (doto (java.security.Signature/getInstance "SHA256withRSA")
           (.initSign private-key (java.security.SecureRandom.))
           (.update msg-data))
         (.sign)
         (.encodeToString (java.util.Base64/getEncoder)))))

borkdude12:01:38

https://github.com/babashka/tools-deps-native: use tools deps from babashka using a pod v0.1.0: upgrade to tools.deps (rather than tools.deps.alpha) See example in pod registry: https://github.com/babashka/pod-registry/blob/master/examples/tools-deps-native.clj

borkdude14:01:05

💯

❤️ 12
borkdude15:01:42

I have posted a link to babashka babooka on the orange website. Perhaps we could hit the front-page today with @nonrecursive’s awesome work? 🚀 🙏

10
borkdude18:01:06

On the front page now, !!!

Darrick Wiebe17:01:46

Hey, can the https://github.com/babashka/pods library be used directly from JVM Clojure? I just learned about pods from @nonrecursive’s post today and it seems nice, simple and flexible. I've got a project that uses libpython-clj just to get the Python AST of a source file, turn it into data and pull it into Clojure for my nefarious purposes. This works great except that I want to distribute my project eventually and it seems hard to stand up on different environments, pinning me to both a specific JVM and a specific Python version, with the need to specify the path of a dll that in practice is not too easy to find on different platforms. It seems that this would be a good alternative strategy, at least worth testing. I saw the Python example in the repo which is nice and simple. What do you think about using transit as the data format? I'd like to keep as much datatype precision as possible...

Darrick Wiebe18:01:36

Hmm I'm actually not sure whether Transit is really supported anymore... Seems broken in Python since 3.3...

borkdude18:01:12

transit is supported for pods and yes, the pods library works on the JVM as well

Darrick Wiebe19:01:33

Perfect, thanks