I have a query that returns events in a particular order allowing me to (partition-by :match-id) to get the events in nice workable batches. Previously, this query was running against athena and the results were generated by an iteration and I was able to generate a lazy sequence and give that to the partition-by call. Now I’m trying to do the same thing with jdbc.next. plan gives you an IReducible right? Is there a way I can take that reducible, and return a lazy sequence of the rows in batches, partitioned by some column in each row (assuming the rows are ordered by that column in the query).

Hm, I thought this might work…

(transduce
   (partition-by :match_id)
   (fn
     ([] nil)
     ([acc] acc)
     ([acc m]
      (prn :processing-match (:match_id (first m))
           :first-row (first m))))
   (jdbc/plan ds ["select match_id, match_level_id
                     from read_parquet('/tmp/arqam-01.parquet')
                    where match_id in (1119060, 1119640)
                    order by match_id, match_level_id"]))

:processing-match 1119640 :first-row {:match_id 1119640, :match_level_id 1}
:processing-match nil :first-row {row} from `plan` -- missing `map` or `reduce`?

Hm, this actually works as expected if I pass a literal vector as the reducible.

(transduce
   (partition-by :match_id)
   (fn
     ([] nil)
     ([acc] acc)
     ([acc m]
      (let [events (map #(select-keys % [:match_id :match_level_id]) m)]
        (prn :processing-match (:match_id (first events))
             :first-row (first events)))))
   [{:match_id 1 :match_level_id 1}
    {:match_id 1 :match_level_id 2}
    {:match_id 2 :match_level_id 1}
    {:match_id 2 :match_level_id 2}])

:processing-match 1 :first-row {:match_id 1, :match_level_id 1}
:processing-match 2 :first-row {:match_id 2, :match_level_id 1}
:processing-match 1 :first-row {:match_id 1, :match_level_id 1}
:processing-match 2 :first-row {:match_id 2, :match_level_id 1}