Fork me on GitHub
#aws
<
2022-10-01
>
Drew Verlee22:10:15

I'm trying to make my aws cloudformation experience better. The obvious way to do that is to code cloudformation in clojure and have a way to get function documentation quickly as well as specs on required fields. So there is some prior art in this regard from https://github.com/brabster/crucible. for example

(def simple (template "A simple sample template"
                      :my-vpc-cidr (parameter)
                      :my-vpc (ec2/vpc {::ec2/cidr-block (xref :my-vpc-cidr)})
                      :vpc (output (join "/" ["foo" (xref :my-vpc)]))))
What do people think of that idea? Anyone with first hand experence? Why have the output produce Refs instead of just inline the value, it's not like it forces you do to that but i'm not sure why its useful to use the cf language at all instead of the static output. Maybe you can't always make a reference via an ID?

Drew Verlee22:10:19

There is also https://github.com/Stedi/cdk-clj which was abandoned. > After developing and using this Clojure wrapper for https://github.com/aws/aws-cdk#aws-cloud-development-kit-aws-cdk for the past six months, we've decided to use TypeScript and AWS' library directly.

Drew Verlee22:10:47

What's interesting is that the docs are obviously machine generated some structure. e.g https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html So if compiler could get that, it should be possible to generate clojure code from it.

Drew Verlee23:10:42

Another interesting tid bit is that cloudformation and the aws api seem to be subtly different and i'm not sure why:

;; Create Role API call

;; {:Path string,
;;  :RoleName string,
;;  :AssumeRolePolicyDocument string,
;;  :Description string,
;;  :MaxSessionDuration integer,
;;  :PermissionsBoundary string,
;;  :Tags [:seq-of {:Key string, :Value string}]}


;; cloudformation for aws iam role
;; {"Type" : "AWS::IAM::Role",
;;  "Properties" ​ {"AssumeRolePolicyDocument" ​ Json,
;;                 "Description" ​ String,
;;                 "ManagedPolicyArns" ​ [String, ...],
;;                 "MaxSessionDuration" ​ Integer,
;;                 "Path" ​ String,
;;                 "PermissionsBoundary" ​ String,
;;                 "Policies" ​ [Policy, ...],
;;                 "RoleName" ​ String,
;;                 "Tags" ​ [Tag, ...]}}


;; things cloudformation has the api doesn't
;;                 "ManagedPolicyArns" ​ [String, ...],
;;                 "Policies" ​ [Policy, ...],

Drew Verlee23:10:51

except thats what the cdk-clj was and they gave up...

Drew Verlee23:10:28

Though it's possible things have improved sense then with the new version I mean aws has to invent lisp eventually, how else are the going to manage all these non lisp languages. https://docs.aws.amazon.com/cdk/v2/guide/home.html

jjttjj01:10:23

For your first point about improving dev experience, have you checked out the cloud formation json spec at all? https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html I messed with a nested search of that once, where I had a rough/quick function that could take a search string for a resource type and spit out a rough skeleton of the resource template.

Drew Verlee01:10:45

I hadn't looked at it, that looks promising.

jjttjj01:10:21

I've seen the cdk java libs used directly from clojure too, which seems pretty nice. I have preferred so far just sticking with cloud formation templates, but cdk had stuff that is much higher levels and for certain types of things it seems nicer

Drew Verlee01:10:55

interesting, yeah, i'm curious if there are any easy wins here. Something tells me the answer is "no" but maybe things have changed sense the last time someone tried.

jjttjj01:10:27

I think a lot of it might depend on what you're trying to do. Lately I've been working primarily with lambdas and have settled on using the serverless transform macro in cloud formation to use this https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html (without using any of the rest of the serverless stack or the sam cli) which has been pretty nice for that

jjttjj01:10:06

But I've also tried to get a fairly minimal fargate example working using cfn, and it wasn't pleasant at all, for that cdk seems much more useful

Drew Verlee01:10:56

i'm suprised aws doesn't have a "how to build a cdk" section for xyz language.

Drew Verlee01:10:10

i guess they don't want to expose... something

jjttjj01:10:18

You mean like how to build a higher layer library on top of cloud formation?

jjttjj02:10:23

What kind of resources are you typically putting together in the stuff you work with?

Drew Verlee02:10:43

Iam, s3, cloudfront, recorsets, coudepipeline, route53

jjttjj02:10:03

> Another interesting tid bit is that cloudformation and the aws api seem to be subtly different and i'm not sure why: One thing about this is the cloudformation templates need to be able to be diffed with the previous template in such a way that aws can magically make your old stack look like the new template in a sensible way, while the api just needs to say "make this thing happen". It's a subtle difference and I can't think of any great examples at the moment that make it clear why they're different, but you'll hit them eventually

👍 1
Drew Verlee02:10:44

yeah, that makes perfect sense. I was thinking i might be able to use the api fn signatures, but it's probably not worth it if there are even slightly different.

jjttjj02:10:38

> Iam, s3, cloudfront, recorsets, coudepipeline, route53 Gotcha yeah I'm just starting to work with some of these but can't speak to them much from experience. I think I saw from one of the earlier discussions here that a lot of people here use cdk or terraform, which might have a better story for those than raw cfn. Personally my tendency is to try to stay in raw cfn but it definitely has its limits.

Drew Verlee02:10:30

i'm just fing shocked that we live in a world where i can ask a machine to generate a picture of our president fighting a dinosaur. And at the same time, i'm coding in YAML. It's like someone looked at the last 3 decades of pl research and wisdom and was like nah i don't need any of that.

jjttjj02:10:36

For example, this is kinda where I drew the line before with cfn, fargate: https://github.com/1Strategy/fargate-cloudformation-example/blob/master/fargate.yaml It's a massive template where you're working with pretty low level stuff compare to what you want to do with the deploying an image, and that template doesn't even include part you need like the VPC/subnets/etc which are in parameters (not my git repo just an example of what it looks like)

👀 1
jjttjj02:10:07

but then the AWS::Serverless::Function I linked above on the other hand is pretty much just an all-in-one resource that will basically be sort of like a deployment config file for your function and is pretty nice to work with

Drew Verlee02:10:28

I'll take a look when i get back, have to do life stuff. i'm currently assuming i'm going to be sticking with raw cf yaml for a while. 😢

jjttjj02:10:32

It might just be a personal thing, but i really dislike yaml and would say that switching to edn then converting to json would be a big quality of life improvement, but i realize this might not be possible or ideal in all real world situations for various reasons

Drew Verlee02:10:35

Oh yeah, that's something we might be able to do. Alternatively, maybe just use typescript cdk.

Leaf Garland04:10:56

We looked at clj-cdk a while back but felt that a lot of the benefits of the cdk are lost unless you use a language with good completion and docs. So we went with typescript cdk. Clojure lsp is loads better these days so I can imagine that a better experience is possible, perhaps with spec to help validate.

Drew Verlee04:10:57

@U02EP7NKPAL the issue is aws doesn't seem to provide a full spec for cloudformation in a machine readable way. Given they have reimplemented it in 5 languages, they either have such a thing, or is a mad house over there. I suspect the later.

Leaf Garland07:10:26

Oh they do have https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html. Also all the different language versions are partly generated and partly wrappers around the typescript SDK - when you use the java CDK lib it https://github.com/aws/jsii. Checkout https://github.com/aws/aws-cdk/tree/main/packages/@aws-cdk/cfnspec to see how they use that data in CDK.

Drew Verlee03:10:38

@U02EP7NKPAL i really appreciate you relinking that, i didn't read it right the first time, now i see the information is in there! Let see what i can do.

excited 2
Drew Verlee05:10:58

(->> cloud-formation-resource-specification
     :ResourceTypes
     :aws.iam/role
     :Properties
     keys
     )
;; => (:ManagedPolicyArns
;;     :PermissionsBoundary
;;     :Path
;;     :Policies
;;     :Tags
;;     :RoleName
;;     :MaxSessionDuration
;;     :Description
;;     :AssumeRolePolicyDocument)


(->> cloud-formation-resource-specification
     :ResourceTypes
     :aws.iam/role
     :Properties
     :RoleName)
;; => {:Documentation
;;     "",
;;     :PrimitiveType "String",
;;     :Required false,
;;     :UpdateType "Immutable"}

Drew Verlee05:10:26

(ns drewverlee.aws.convert-cloud-formation-resource-specification
  (:require
   [hickory.core :as h]
   [clojure.string :as str]
   [clojure.data.json :as json]
   [camel-snake-kebab.core :as csk]))

(defn aws-resource-string->aws-resource-keyword
  [aws-resource-string]
  (let [ss (map csk/->kebab-case (str/split aws-resource-string #"::|\."))]
    (keyword
     (str
      (str/join "."  (butlast ss))
      "/"
      (last ss)))))

(def cloud-formation-resource-specification
  (-> "/home/drewverlee/Personal/try-cloudfromation/src/CloudFormationResourceSpecification.json"
      slurp
      (json/read-str :key-fn (fn [k] (if-not (str/includes? k "::")
                                       (keyword k)
                                       (aws-resource-string->aws-resource-keyword k))))))

Drew Verlee05:10:46

ill push to a github repo in a second.