Fork me on GitHub
#babashka
<
2021-01-11
>
steveb8n01:01:44

Q: I’m planning to convert my CI aws bb scripts from using shell calls to use the new aws pod. I’d like to start using Vault at the same time https://github.com/99designs/aws-vault

steveb8n01:01:12

has anyone done this i.e. tested that the creds provider works with Vault?

borkdude08:01:48

@U0510KXTU I'm not sure about the details of this by @U0FT7SRLP could probably tell you more

borkdude08:01:16

What is always possible is to use the credential_process provider to shell out to some custom script which connects to custom providers

steveb8n08:01:49

thanks. good to know that there’s a programmatic way to do this. since Vault is considered a creds best practice, eventually the bb pod will support it

steveb8n08:01:04

hopefully it just works 🙏

borkdude08:01:54

Maybe also good to post an issue

steveb8n09:01:36

I’ll give it a try and will log an issue if it doesn’t work. otherwise I’ll provide some doc for you if it needs some extra config

jeroenvandijk09:01:50

@U0510KXTU I haven’t used aws-vault and I’m not familair with it’s features. But I assume it can be made compatible with credential_process . Can the tool print temporary credentials to stdout?

steveb8n09:01:15

no idea. it’ll be the first time I’m using it as well

steveb8n09:01:34

but I was advised by AWS architects that I should be using it

steveb8n09:01:52

so I’ll report back. thanks for replying 🙂

jeroenvandijk09:01:53

Ok 🙂 They say a lot of things 😅

borkdude09:01:00

you can always make it print the creds in JSON using a custom script

steveb8n09:01:30

dat klopt 😉

steveb8n09:01:03

that said, since I’m providing enterprise SAAS I need to be able to show best practice security so I’ll follow advice like this

jeroenvandijk09:01:39

I’m joking, the aws-fault is probably a good practise. Just not the only one. @U0JEFEZH6 is using AWS SSO. I think that could be an even better practise

steveb8n09:01:58

yeah, I’m also using a yubikey

jeroenvandijk09:01:00

I’m using Okta (similar to AWS SSO)

steveb8n09:01:20

but still need to configure assumed roles. so many layers!

jeroenvandijk09:01:34

yeah ignorance is bliss

💯 4
steveb8n09:01:41

Okta was founded by my former manager. I know it well

steveb8n09:01:39

</end> name dropping

jeroenvandijk09:01:45

Cool! I started using it before there was AWS SSO and haven’t tried that yet. So I can not make a proper comparision. I’m assuming AWS SSO is a lot easier to use with AWS. Had to build my own tool for Okta

steveb8n09:01:06

I’m using Cognito for my users. It’s not as nice as Auth0 etc but it removes an external dependency. I’d bet AWS SSO is similar. Okta is bound to be better since it’s their specialty

steveb8n09:01:54

Thanks for the feedback. I’ll report back

👍 4
lukasz14:01:22

I think you can now wire things up a bit differently with Okta and AWS SSO - meaning, that Okta is just a SAML 2.0 idP (we use Google G-Suite/Workspace/Apps), and from there it doesn't really matter how you log in after running aws sso login --profile=.... - my team just has to copy the aws config from our repo, with all of the required profile entries (account IDs, assumed roles etc) and works out of the box. I have never used aws-vault and Okta's own AWS auth (I believe it's a thing) so cannot comment on that

nate17:01:05

looks like aws-vault exposes creds as environment variables, so the babashka pod should just work?

lukasz17:01:07

there's a similar tool for AWS SSO https://github.com/linaro-its/aws2-wrap (I rewrote it in Clojure as a custom auth provider for aws-api)

jeroenvandijk14:01:43

@U0JEFEZH6 I added caching to the aws sso credential_process script. So ready for usage I think 🙂 It also works with the aws cli https://gist.github.com/jeroenvandijk/ace7432be94d083e63729ac313a0b78f

lukasz15:01:42

@U0FT7SRLP really cool - if I understand correctly, most SDKs can pick up the credential_process option, including bb's aws pod?

jeroenvandijk16:01:35

@U0JEFEZH6 yes, from experience I know this to be true for the aws cli (so maybe also python skd’s) and the java sdk (used by Amazonica). aws-api doesn’t support it yet (https://github.com/cognitect-labs/aws-api/issues/73)

lukasz16:01:45

Awesome, that's super helpful

👍 3
borkdude16:01:17

@U0FT7SRLP when you're done developing this, maybe we could add it to the docs of the pod

borkdude16:01:32

or to an examples dir

jeroenvandijk16:01:44

Yeah I was thinking of how to distribute this in a convenient way. I have more scripts like this in mind, so was actually thinking of a seperate repo. Maybe even installable via homebrew or something so it is easy to find and install

jeroenvandijk16:01:18

Also it becomes complex pretty quickly, so tests would be nice too

borkdude16:01:39

well, maybe as a gitlib?

jeroenvandijk16:01:31

could be helpful when using it as a library. gitlib’s don’t support adding binaries to the PATH, right?

borkdude16:01:14

well, you can check anything into git, but maybe not the best thing to do. why?

borkdude16:01:56

what kind of binary?

jeroenvandijk16:01:10

just to make it a bit more convenient. Like how you install babashka through homebrew and not having to manage everything yourself (including updates)

jeroenvandijk16:01:27

a binary as in chmod +x script/my-babashka.clj

lukasz16:01:11

btw, I'll most likely write my own version of the SSO provider, as for example - when configuring SSO you have to have a valid config with profiles - @U0FT7SRLP version assumes that some of that is configured in source

👍 3
jeroenvandijk16:01:13

I’ll look into it later. For now, I spend way too much time on everything related to this pod 😅

borkdude16:01:07

but this was also the pod / native aws thing you have tried to come up with for a long time... right? so not time wasted...?

💯 3
jeroenvandijk16:01:09

@U0JEFEZH6 My script only assumes you give it the right arguments in your credentials file (region, account-id, role and start url).

jeroenvandijk16:01:06

@U04V15CAJ sorry definitely didn’t mean wasted! Just that I need to focus a bit on other things and not sure when I’ll get to wrapping all these other things properly

borkdude16:01:14

@U0FT7SRLP if you are distributing .clj files, I think gitlibs are the way to go.

(require '[babashka.deps :as deps])
(deps/add-deps '{:deps {jeroenvandijk/scripts {:git/url "..." :sha "..."}}})
(require '[jeroenvandijk.scripts.aws-thing :as aws-thing])

jeroenvandijk16:01:52

Yeah looks good! Thank you

borkdude16:01:09

And we have already 5 contributors now in such a short time. I think it's a success. Thanks for the time you put into the credential stuff especially, the gnarliest bit :)

jeroenvandijk16:01:33

I have to thank for you sure! 🙏 For putting that much time in this and not even being a real AWS user (yet, right?)

borkdude16:01:09

Right, but we're moving to the cloud now at work too, so I'll probably be one soon (either AWS, or Azure)

jeroenvandijk16:01:20

I’m also happy with the result. I have spent a lot of time in getting AWS tools to compile with Graalvm. This will not be the last one I think

jeroenvandijk16:01:43

Ah nice. Good to hear you can start with Clojure tools from the start (when using AWS)

roklenarcic10:01:16

this is more related to sci, but can you interrupt (timeout) a sci.core/eval-string ?

borkdude10:01:41

@roklenarcic you can do this by using the same strategy as nREPL does this: run it in a thread and use the (deprecated!) Thread#stop method

roklenarcic10:01:52

In Java the new, non-obsolete, method for doing this is to check Thread/interrupted at various parts and end if it is so. Thread/stop can fail to free some resources. For instance in an interpreter like sci, it would be easy to check Thread/interrupted after each expression is evaluated.

roklenarcic10:01:47

Future stopping and ExectorService shutdown also sends interrupted signal to the tasks

borkdude10:01:26

This still won't catch things like (doall (range)) since it would require those functions to adopt the same strategy

borkdude10:01:37

in clojure.core

lgouger13:01:35

Hoping this is useful and also looking for feedback.  paging-invoke a function to lazily call aws/invoke to page through large result sets. https://gist.github.com/lgouger/2262e2d2503306f2595e48a7888f4e73

metal 10
lukasz14:01:35

Another huge shoutout to bb - I was able to convert my legacy "full" Clojure project, which interacted with BigQuery, to just one file bb script (which uses bb.process and bq command line tool).

🎉 28