Is it possible to lazily iterate over a big result set, invoking a function on each row?

the answer will differ depending on which layer you want to do this in; in postgres you can do this with cursors

So, I am using postgres. But the function I'd want to invoke on each row would be a clojure function. Was hoping there might be a way to do this using clojure.java.jdbc

@cddr this may help: https://stackoverflow.com/questions/39765943/clojure-java-jdbc-lazy-query

See the docs under Processing a result set lazily here http://clojure-doc.org/articles/ecosystem/java_jdbc/using_sql.html

Aha perfect. Thanks!

@cddr Pay attention specifically to the caveats around database-specific tweaks you may need to make the result set processing fully lazy -- auto-commit on opening connections, result set cursors etc.

@cddr The reducible-query machinery is relatively new, but is the recommended way to deal with processing large result sets. If you hit any issues with it, let me know. We use it in production at work, but not extensively yet.

Cool thanks! I shall give it a whirl

I've added another answer there and gone through a few other SO posts relating to clojure.java.jdbc. Clearly, this is an ongoing project to try to keep answers up-to-date and accurate 😞