Fork me on GitHub
#sql
<
2022-05-13
>
seancorfield00:05:44

(run! row-processor (jdbc/plan ds [my-sql])) -- where the row-processor could just pull the columns from the row and do whatever's needed without worrying about a result (and it wouldn't need to care about qualified names since they don't exist inside plan: it could even use strings at that point since it's going against JDBC directly, e.g., (run! #(do-stuff (get % "column")) (jdbc/plan ...)) or (get % :column) if you prefer, but it just calls name and then passes that string directly to JDBC to fetch the column (by label).

Jim Strieter14:05:12

"And if you only want to do a single pass over the results and process each row, perhaps plan would be better and then you don't even have the overhead of creating the hash maps or the vector?" I like that idea. How do I do it?

seancorfield15:05:16

@jimstrieter Have you read through the docs? There are several examples.

seancorfield15:05:54

In particular, plan lets you process result sets that are larger than will fit in memory, effectively streaming results "lazily" from the database. Everyone seems to jump straight to execute! but I try to encourage everyone to use plan for heavy lifting instead.

seancorfield15:05:12

Think of plan as returning a "virtual collection" -- it's a reducible (an IReduceInit in fact, but it is also foldable if you want to use clojure.core.reducers and do processing in parallel with fold). Nothing actually happens until you reduce the result of plan: then it gets the Connection, initiates the SQL operation, and then your reducing function is invoked on each row, passing in a very thin wrapper around a ResultSet that supports the same operations as a Clojure hash map but doesn't actually create a Clojure data structure unless you perform an operation that requires one, such as assoc, seq, count, etc. Lookups are done directly against the ResultSet object.

seancorfield15:05:11

And when the reduction is complete -- either at the end of the ResultSet or through early termination via reduced -- the Connection is closed/returned to the pool. Automatic resource management.