sql

jmckitrick 2023-04-14T12:22:14.267819Z

Has anyone ever tried to use the copy command from a stream inside clojure? In other words, generating the data programmatically and streaming it to the command, instead of to a file and then reading it into the copy command. I see the STDIN option, but not sure how to wire that into a Clojure stream, or if it’s even possible.

kolstae 2023-05-16T11:47:03.910559Z

From the Clojure deref I saw this https://github.com/igrishaev/pg Haven't tried it yet, but I will

kolstae 2023-04-14T14:36:40.816499Z

Yes, I've done this. I'ts a bit awkward but invaluable for large amounts of data

jmckitrick 2023-04-14T14:37:18.190019Z

So you did it without using the filesystem, just streams within clojure?

kolstae 2023-04-14T14:39:45.019729Z

I use core.async to deliver each line

jmckitrick 2023-04-14T14:41:33.244509Z

That looks like what I need. I assume I could use a lazy sequence in place of the async call.

jmckitrick 2023-04-14T14:42:34.560019Z

The 2 PG classes look important as well. Where can I find those?

jmckitrick 2023-04-14T14:42:50.115009Z

Any jdbc for postgres?

kolstae 2023-04-14T14:44:05.784539Z

Yeah, just replace the loop with doseq perhaps. The imports are:

(org.postgresql PGConnection)
(org.postgresql.copy PGCopyOutputStream)

kolstae 2023-04-14T14:44:41.805999Z

... and they are part of the pg-jdbc jar

jmckitrick 2023-04-14T14:45:12.252869Z

Perfect. I’ll give them a try!

jmckitrick 2023-04-14T14:47:35.016459Z

Thanks!

kolstae 2023-04-14T14:48:11.959739Z

🙂

igrishaev 2023-10-10T08:26:40.496729Z

@jmckitrick It's possible now with pg 0.1.9, please check out the docs: https://github.com/igrishaev/pg/blob/master/doc/090-copy.md

igrishaev 2023-10-10T08:28:23.288409Z

Briefly, there are a couple of functions that take a seq of rows/maps and send them to the database without having an intermediate stream

jmckitrick 2023-10-10T10:07:09.795969Z

@igrishaev Excellent, thank you!

jmckitrick 2023-05-19T11:21:05.991749Z

I saw that as well. Bookmarked!