So, I have a setup where multiple SQL queries are executed in parallel, and we need to stream the results via a CSV formatter, then zip the resulting files, then upload to S3. All must be done simultaneously, with the upload starting as soon as the first bytes are zipped, and with the zipping starting as soon as the first CSV data appears, and the CSB formatting starting as soon as the data begin to come lazily from the db. For these purposes, we use Akka streams, calling Scala from Clojure. Akka makes sure that the data in the graph flows smoothly, with a backpressure-handling protocol, so that the actors only request the next data from a producer asynchronously when the consumer has the capacity to process it. As a result, the memory isn't supposed to grow, no matter how big the data returned by the queries is, only a slice of it loaded into memory at any given time (like a river that transports a lot of water without overflowing its banks). However, we see a problem at the db end, which isn't Akka. We use jdbc.core to query the database to produce a cursor, which is transformed into a lazy seq, then pass the iterator of the lazy seq to Source/fromIterator to create an Akka source (a node in the flow graph). Something like this:

(let [cursors (mapv #(jdbc.core/fetch-lazy connection % opts) sql-stmts) ;; could be multiple connections, too, depending...
      records (mapv #(.iterator (jdbc.core/cursor->lazyseq %)) cursors)]
  (...some code...))

You say lazy seqs, but are using mapv which produces a vector which is not lazy

You might be running into a chunking issue, seqs produced from vectors are chunked, and depending may be processed 32 elements at a time instead of one at a time

If by jdbc.core you mean (https://funcool.github.io/clojure.jdbc/latest/api/jdbc.core.html) I would suggest moving away from that. It started as a clone of the clojure contrib library clojure.java.jdbc, and there is some bad blood there about the author of clojure.jdbc copy and pasting code from clojure.java.jdbc

The primary maintainer of clojure.java.jdbc has released a next generation clojure jdbc library next.jdbc which addresses a lot of short comings in clojure.java.jdbc (and presumably in clojure.jdbc as well)

Ah, good points. Thanks.

The next.jdbc library is built from the ground up around using reduce for processing results, which can be better for resource usage

But that mapv just creates a vector of lazy seqs, not a vector of vectors. Well, a vector of iterators for lazy seqs.

The problem with plan in next.jdbc is that, for my purposes, I don't want a reducible as a result. I need an iterator, so that it could be queried asynchronously.

A reducible can't be passed to Akka.

How about just using prepare, then processing the rows manually? Maybe Clojure makes it too hard to control the execution precisely when using laziness, like hiredman was talking about. Could be a bug in your code or the library you are using. A similar option would be to reduce a plan for side-effects (like populating a queue or channel) if reading manually is too cumbersome.

But the side-effects wouldn't wait for the consumer's capacity to meet demand. The entire point is to delegate backpressure handling to the Akka framework.

jdbc just isn't, uh, I guess push based would be the way to descibre it, with jdbc you pull the results from the database. there have been sort attempts to make something more asynchronous but nothing that has been standardized in the jdbc interface. I believe oracle was even working on it at one point, but decided not to bother and wait for project loom