sql 2023-04-14

Has anyone ever tried to use the copy command from a stream inside clojure? In other words, generating the data programmatically and streaming it to the command, instead of to a file and then reading it into the copy command. I see the STDIN option, but not sure how to wire that into a Clojure stream, or if it’s even possible.

From the Clojure deref I saw this https://github.com/igrishaev/pg Haven't tried it yet, but I will

Yes, I've done this. I'ts a bit awkward but invaluable for large amounts of data

So you did it without using the filesystem, just streams within clojure?

I use core.async to deliver each line

That looks like what I need. I assume I could use a lazy sequence in place of the async call.

The 2 PG classes look important as well. Where can I find those?

Any jdbc for postgres?

Yeah, just replace the loop with doseq perhaps. The imports are:

(org.postgresql PGConnection)
(org.postgresql.copy PGCopyOutputStream)

... and they are part of the pg-jdbc jar

Perfect. I’ll give them a try!

@jmckitrick It's possible now with pg 0.1.9, please check out the docs: https://github.com/igrishaev/pg/blob/master/doc/090-copy.md

Briefly, there are a couple of functions that take a seq of rows/maps and send them to the database without having an intermediate stream

@igrishaev Excellent, thank you!

I saw that as well. Bookmarked!