plan is intended to be used with reduce or transduce or similar -- it produces a reducible collection, but doesn't actually do anything until you reduce it.

See the examples in the documentation.

what Im failing to understand is: is it possible to iterate through the results of a plan (just to do IO, I dont actually need to reduce anything) without loading the entire collection into memory? maybe plan is not right function to use? my use case is simple, I want to iterate through the results of plan and write each row into a stream, without loading the entire collection in memory maybe there is a better function to achieve this?

reduce

You have to use reduce. (or similar).

plan's "result" is a reducible collection.

execute! produces a vector with the entire result set in it. If you want to "stream" a result set, you have to use plan and reduce.

alright, reduce should do it, thanks a lot for the detailed explanation 🙂

There is no lazy (Clojure) interface to the data in next.jdbc -- deliberately 🙂

You'll also want to read the Tips & Tricks section of the docs for your database for any hints around how to get the JDBC driver to stream result sets properly -- it's DB-specific, unfortunately.

(MySQL and PostgreSQL have documentation there for streaming result sets)

is it possible to get the column names of a result-set before reducing it? or should I reduce the first element and figure out manually?

naively tried to execute (jdbc.result-set/get-unqualified-lower-column-names (.getMetaData plan-result) {}) but it could not find the method

As I keep saying, the result of plan is a reducible collection -- the only thing you can do with it is reduce it.

Inside the reducing function, you can call column-names on the "row" abstraction. You can also call metadata on it. In general, with plan, you can only operate on the "row" abstraction (which is a thin map-like wrapper over the underlying mutable ResultSet), inside the reducing function.

This shows what's available on that abstraction: https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.3.847/api/next.jdbc.result-set#InspectableMapifiedResultSet

Also, inside the reducing function, there is no notion of qualified column names -- they are column labels. There is no hash map (in the Clojure sense). The "row" is a view onto the ResultSet object, so (:foo row) fetches the column labeled "foo" from the current ResultSet's row.

got it! thanks for the clarifications again!

Thinking about this some more, there's kind of a big gap between the "easy" stuff with next.jdbc -- execute! and the "friendly SQL functions" which all traffic in Clojure data structures -- and the "simple" but more advanced stuff in the library, where you need to have more of an understanding of the underlying JDBC stuff as well as Clojure stuff like reducible collections/reducing functions/abstractions/protocols/etc. If you have any suggestions for how to provide a "bridge" between those in the documentation, that would have made it easier for you to figure out how to solve your initial problem, I'd be happy to update/expand the docs! I'm a bit too familiar with all that to really have a good feel for what needs additional explanation in the next.jdbc docs as opposed to pointing folks at more JDBC docs and/or more Clojure docs.

I think the major problem for me was I never used the reduce fn to do side fx, and it took me a while to understand that whatever I do it should be inside of a reduce also the confusion with lazyness

Hmm, maybe I should add some examples to the docs using run! for when you want to process a plan for side effects and throw away the results? run! is literally a reduce expression that throws away the result:

build=> (source run!)
(defn run!
  "Runs the supplied procedure (via reduce), for purposes of side
  effects, on successive items in the collection. Returns nil"
  {:added "1.7"}
  [proc coll]
  (reduce #(proc %2) nil coll)
  nil)

https://github.com/seancorfield/next-jdbc/issues/236

yeah that might help, it just felt wrong using reduce for that, and it turns out run! is more suitable for my needs thanks for pointing out its existence