Fork me on GitHub
#aws
<
2020-02-01
>
ghadi19:02:59

We are considering adding a new function to Clojure called iteration, and it would be very useful for consuming paginated APIs like Amazon's

(defn log-groups
  "gets all CloudWatch Log Groups"
  [client]
  (->> (iteration
        (fn [token]
          (aws/invoke client (cond-> {:op :DescribeLogGroups}
                               token (assoc :request {:nextToken token}))))
        :kf :nextToken
        :vf :logGroups)
       (into [] cat)))
^ here's how it could work with Cognitect Labs' aws-api

ghadi19:02:44

(still WIP, reserve the right to not do it, ymmv, etc.)

kenny19:02:31

Nice. First thought is how do I handle anomalies? Sometimes (most times) I'd like to essentially have a reduced if an anomaly occurs, returning the anomaly.

ghadi19:02:42

the iteration above will terminate on anomalies (because :kf won't return anything from an anomaly) -- the (into [] cat) is swallowing it though

ghadi19:02:02

you take out :vf or pass in a smarter :vf

ghadi19:02:26

then could check for the anomaly on the outside

ghadi19:02:19

(into [] (halt-when anomaly? retf) (iteration....))

ghadi19:02:50

halt-when will return the failing input when the retf isn't given

kenny20:02:12

Hmm, I see. Clever. Looking at the pagination code we are using, the only other thing we keep track of is page-count. In some cases we use that for logging when certain page counts are hit for long running iterations & other cases for capping pages followed. Seems like you can do that here by returning a different data structure from :kf. I think this would cover our use case.

ghadi20:02:13

probably better to pass in a smarter :vf (for the anomaly case)

kenny20:02:29

The only other comment is I almost always prefer apis where the function takes its options in a map instead of the variadic approach. Easer to programmatically build up when needed. Was there a reason you chose the variadic args?

ghadi20:02:08

I suspect that this isn't going to be a case where calls are programmatically built up, but I'll take that feedback back.

ghadi20:02:26

my suggestion for halt-when isn't useful for (into []) but is for transduce

ghadi20:02:05

user=> (transduce (halt-when odd?) (completing conj! persistent!) (transient []) [2 2 2 3])
3
user=> (into [] (halt-when odd?) [2 2 2 3])
Execution error (ClassCastException) at user/eval197 (REPL:1).
class java.lang.Long cannot be cast to class clojure.lang.ITransientCollection (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.ITransientCollection is in unnamed module of loader 'app')

kenny20:02:57

That’s far less clean looking

ghadi20:02:35

if into was redefined slightly it would be possible

kenny20:02:52

It does seem like into should support halt-when.

ghadi20:02:23

(defn into'
  ([to xform from]
   (let [add-meta #(with-meta % (meta to))]
     (if (instance? clojure.lang.IEditableCollection to)
       (transduce xform (completing conj! (comp add-meta persistent!)) (transient to) from)
       (transduce xform (completing conj add-meta) to from)))))

ghadi20:02:32

redefined like that ^ it supports it

ghadi20:02:56

user=> (into' [] (halt-when #{:anomaly} (fn [& args] (zipmap [:result :failing-input] args))) [1 2 3 :anomaly 4])
{:result [1 2 3], :failing-input :anomaly}

kenny20:02:31

Any idea if there’s a reason into wasn’t written to support halt-when from the start?

ghadi20:02:16

if that problem is valid, it's an oversight or bug