Fork me on GitHub
#sql
<
2018-05-29
>
cddr13:05:29

Is it possible to lazily iterate over a big result set, invoking a function on each row?

lady3janepl13:05:19

the answer will differ depending on which layer you want to do this in; in postgres you can do this with cursors

cddr13:05:09

So, I am using postgres. But the function I'd want to invoke on each row would be a clojure function. Was hoping there might be a way to do this using clojure.java.jdbc

seancorfield16:05:06

Those answers are horribly outdated and misleading, unfortunately.

cddr15:05:13

Aha perfect. Thanks!

seancorfield16:05:41

@cddr Pay attention specifically to the caveats around database-specific tweaks you may need to make the result set processing fully lazy -- auto-commit on opening connections, result set cursors etc.

seancorfield16:05:31

@cddr The reducible-query machinery is relatively new, but is the recommended way to deal with processing large result sets. If you hit any issues with it, let me know. We use it in production at work, but not extensively yet.

cddr16:05:59

Cool thanks! I shall give it a whirl

seancorfield17:05:06

I've added another answer there and gone through a few other SO posts relating to clojure.java.jdbc. Clearly, this is an ongoing project to try to keep answers up-to-date and accurate 😞