Has anyone ever tried to use the copy command from a stream inside clojure? In other words, generating the data programmatically and streaming it to the command, instead of to a file and then reading it into the copy command. I see the STDIN option, but not sure how to wire that into a Clojure stream, or if it’s even possible.
From the Clojure deref I saw this https://github.com/igrishaev/pg Haven't tried it yet, but I will
Yes, I've done this. I'ts a bit awkward but invaluable for large amounts of data
So you did it without using the filesystem, just streams within clojure?
I use core.async to deliver each line
That looks like what I need. I assume I could use a lazy sequence in place of the async call.
The 2 PG classes look important as well. Where can I find those?
Any jdbc for postgres?
Yeah, just replace the loop with doseq perhaps. The imports are:
(org.postgresql PGConnection)
(org.postgresql.copy PGCopyOutputStream)... and they are part of the pg-jdbc jar
Perfect. I’ll give them a try!
Thanks!
🙂
@jmckitrick It's possible now with pg 0.1.9, please check out the docs: https://github.com/igrishaev/pg/blob/master/doc/090-copy.md
Briefly, there are a couple of functions that take a seq of rows/maps and send them to the database without having an intermediate stream
@igrishaev Excellent, thank you!
I saw that as well. Bookmarked!