Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-04-14

Channels

#sql
<
2023-04-14
>
jmckitrick12:04:14

Has anyone ever tried to use the copy command from a stream inside clojure? In other words, generating the data programmatically and streaming it to the command, instead of to a file and then reading it into the copy command. I see the STDIN option, but not sure how to wire that into a Clojure stream, or if it’s even possible.

kolstae14:04:40

Yes, I've done this. I'ts a bit awkward but invaluable for large amounts of data

jmckitrick14:04:18

So you did it without using the filesystem, just streams within clojure?

kolstae14:04:45

I use core.async to deliver each line

jmckitrick14:04:33

That looks like what I need. I assume I could use a lazy sequence in place of the async call.

jmckitrick14:04:34

The 2 PG classes look important as well. Where can I find those?

jmckitrick14:04:50

Any jdbc for postgres?

kolstae14:04:05

Yeah, just replace the loop with doseq perhaps. The imports are: 

(org.postgresql PGConnection)
(org.postgresql.copy PGCopyOutputStream)

kolstae14:04:41

... and they are part of the pg-jdbc jar

jmckitrick14:04:12

Perfect. I’ll give them a try!

kolstae11:05:03

From the Clojure deref I saw this https://github.com/igrishaev/pg Haven't tried it yet, but I will

jmckitrick11:05:05

I saw that as well. Bookmarked!

igrishaev08:10:40

@U061KMSM7 It's possible now with pg 0.1.9, please check out the docs: https://github.com/igrishaev/pg/blob/master/doc/090-copy.md

igrishaev08:10:23

Briefly, there are a couple of functions that take a seq of rows/maps and send them to the database without having an intermediate stream

jmckitrick10:10:09

@U1WAUKQ3E Excellent, thank you!