Fork me on GitHub
#aws
<
2019-01-03
>
aaelony00:01:50

hi! Does anyone know if there is a programmatic way to discover what AMIs are available in the marketplace? e.g. the identifier of all deep learning AMIs?

aaelony16:01:21

@dottedmag that looks interesting. I will check it out. Thank-you

aaelony18:01:40

hmmm. not sure this is working well just yet, but something along these lines:

(defn find-AMIs
  "Search the `Name` field for a matching string in all available AMIs."
  ;; 
  [regex-string]
  (let [re (re-pattern (str ".*" regex-string ".*"))
        ami-maps (:Images (aws/invoke ec2-client {:op :DescribeImages
                                                  :request {:Filters [{:Public true}]
                                                            ;;:DryRun true
                                                            }}))
        ami-names (->> ami-maps
                      (mapv :Name)
                      (filterv #(not (nil? %))))
        ami-name-matches (->> ami-names
                              (filterv #(re-matches re)))
        ]
    (vec ami-name-matches)
    ))

aaelony18:01:46

the idea is that you could then seek out deeplearning AMI’s with (find-AMIs “deeplearning”) or so

gws19:01:34

you can also use their filter syntax, this one filters on the Description field but you can do it with Name as well: aws ec2 describe-images --owners amazon --filters 'Name=description,Values=*deep learning*' 'Name=state,Values=available' --output json - it should be easy enough to do something like that with the function you have there

aaelony19:01:03

I saw that but I didn’t have time to figure it out just yet. If you have a sense for it, please lmk

aaelony19:01:17

also, I believe the AMIs would need “deep*learning” since deeplearning strings are common

gws19:01:18

Something like: {:Filters [{:State "available"} {:Description (str "*" regex-string "*")}]}

gws19:01:45

yeah, you can do that too

aaelony19:01:48

I’ll give that a try

gws19:01:43

it's a case-sensitive search, so you might still need to fetch all of them and run them through a filter as you've done, just depends on your requirements

aaelony19:01:46

I recall now, I tried what you suggest and it doesn’t return any results

aaelony19:01:00

That’s why I opted to just use clojure’s filterv

aaelony19:01:17

but I think it was an error on my part

aaelony19:01:32

give me a few minutes, I think I can get it working

aaelony19:01:24

interesting. I ran one without the filter and one with the filter. Both return a count of 84214.

aaelony19:01:52

I think there are roughly 116 or so deeplearning AMIs

aaelony19:01:20

…which is what the filterv version returns

gws22:01:57

well feel free to ignore that suggestion if it doesn't work! I kind of want to know why now though, but there's nothing incorrect about the filter solution certainly, unless you're worried about API limits or conservation of network resources

grzm23:01:06

I think I must be missing it, but I don’t see Cloudwatch among the releases for the cognitect aws client api. https://github.com/cognitect-labs/aws-api/blob/master/latest-releases.edn

alexmiller23:01:43

Might be best to file a github issue so this doesn’t get lost - I don’t know off the top of my head

👍 5
grzm00:01:38

.@U0ENYLGTA followed up in the ticket: it’s in the monitoring endpoint

aaelony23:01:56

@gws, I don’t doubt there is a way to make the filter part work in the request. I just haven’t found it yet and the clojure side filter does in fact work. At some point, I hope to be able to devote time to figuring out the request side filter but can’t at the moment (and the workaround suffices)