Hm, this doesn't feel good.

So, I'm looking to migrating to next-jdbc. We have a nice wraper around clojure.jdbc that we use for lazy batch processing(millions of rows, doseq/partition-all based).

I've written it in two different ways with next-jdbc and both feel "unnatural". I first wrote it with next.jdbc/plan + an eduction

(eduction (map transformation-goes-here) (partitiona-all batch-count) (jdbc/plan ....))

Then with channels, dispatching from inside a simple reduce over jdbc/plan (and a lazy-seq on the channel). Neither, I feel, gives the flexibility of a simple "lazy-seq".

Am I missing something obvious or should I get used to transducers as the new reality for cursors?

How were you doing it with clojure.java.jdbc?

plan is a refinement of reducible-query which is what you should have been using already I would expect.

We were using a custom :result-set-fn that was tying into a sink/source abstraction.

I wouldn't say it's equivalent and I am very open to trying the new next-jdbc way.

But does what I wrote above make sense? Are those the two approaches we have with plan?

Then just use reduce over plan with a variant of that function

Yeah, the eduction variant above is pretty much that, no?

I think the two approaches you described are overkill

Hm

I am missing something them. My impression is that if I reduce the plan directly, I can't actually batch.

because the unit is always one row, so I def needs to partition before reducing.

is that incorrect assumption?

The result set fn must be eager by definition

A transducer can be stateful and handle partitioning if you need that

So transduce over plan if you need more than reduce.

Wait, if I understand correctly.

as plan is already reducible, I don't need the eduction?

Hm

Right

And your result set fn before had to be eager

So you're not losing laziness here

But in this case, the ONLY eager fn required is the final reduce, right?

yes.

I think I get it, thanks. Let's see if I can dumb down this a bit in code.

That's just how transducers work : they are eager

ty. btw, I was surprised by how few people seem to have migrated to next-jdbc.

I understand it, somehow. But still, I hope it actually picks up. The API is a lot nicer.

(and still similar to make a migration relatively painless)

I'm actually quite pleased with how fast adoption is going -- given Clojure's slow pace of change in general 😁