Fork me on GitHub

(the nested-tx and multi-rs branches have both been updated to match develop and I will continue working on those features for future releases)

👍 3

seancorfield/next.jdbc {:mvn/version "1.0.476"} -- extends the "default options" behavior to the friendly SQL functions in next.jdbc.sql

Gleb Posobin19:06:20

Is there a way to iterate over the result set lazily in next.jdbc?


@posobin There are two "lazy" things in play in questions like that: lazy streaming of results from the database, and lazy sequence processing of result sets. The former is possible via next.jdbc/plan. The latter is not -- because something has to manage all of the resources around connections and prepared statements etc.


In it was possible to provide a lazy :result-set-fn but the behavior was pretty much always wrong because the lazy sequence escaped the connection context and you nearly always got an exception from attempting to operate on a closed connection!


In next.jdbc, you can use plan and reduce/transduce/etc to process result sets that will not fit in memory, and you can terminate processing early via reduced (see docs for reduce and reduced in clojure.core).


You can use plan for results that would fit in memory too, and you can still control fetch size and other ways to lazily stream the result set. The processing itself must be eager, however. Also, note that persuading some databases to actually stream very large sets is tricky -- and the exact options needed tend to vary from database to database.


@posobin Can you explain your particular use case, if you're looking for a lazy sequence?

Gleb Posobin03:06:59

I wanted to save a couple of million rows to a JSON, in the end I just made a lazy seq that did batched queries to the db.

Gleb Posobin03:06:32

So basically pagination with offsets and limits.


Hmm, so streaming rows into a JSON file? You could do that with reducing over plan by appending the JSON of each row to a file, with trailing , added I guess? (and a leading [ and trailing ])


Reduce is eager. The "lazy" part refers to how the rows are read from the database

Gleb Posobin15:06:46

So the bottleneck for me is in ram, I didn't want to have the whole result set in memory.


Right, so you must "lazily stream" the result sets from the database, but you can eagerly reduce them.


Getting the driver to lazily stream results is database-dependent -- some suggestions are given in the next.jdbc docs, on the Tips &amp; Tricks page as I recall.

Gleb Posobin16:06:48

Ah, that's great, I'll take a look, thank you!

Gleb Posobin12:06:34

Works great, thank you!

Gleb Posobin13:06:50

I should have just tried it instead of googling for that SO question...