Fork me on GitHub
#sql
<
2022-11-06
>
kwladyka11:11:02

Someone have ready solution with next.jdbc for postgresql and COPY with STDIN ? Reading CSV, remove columns, write to STDIN. As I understand binary is the most performant way.

kwladyka11:11:42

Is it possible to do it with next.jdbc ?

kwladyka11:11:36

https://www.npgsql.org/doc/copy.html https://stackoverflow.com/questions/14242117/java-library-to-write-binary-format-for-postgres-copy I found this example. I am not sure if it is possible in next.jdbc just like that. It looks like it needs special support. Can I use at least part of the next.jdbc code or there is really nothing to use in next.jdbc for COPY?

kwladyka12:11:19

I know there is https://github.com/jgdavey/clj-pgcopy but last change was 2 years ago and I would like to use 1 library instead of 2 for postgresql.

kwladyka15:11:35

(org.postgresql.copy CopyManager
                                CopyIn
                                PGCopyOutputStream)
Perhaps it needs this ^ But I am still figuring this out possibilities

kwladyka15:11:45

hmm I think it is a little harder, because with binary it also need data type support

Jan Winkler16:11:01

But COPY is postgres-specific, there is nothing to be done on JDBC level of abstraction and definitely not in next-jdbc, which is "just" a JDBC wrapper... You found the copy API in psql driver and that is what you should use. Either directly via java interop or using clj-pgcopy, if it works with your/current psql driver.

kwladyka16:11:54

perhaps. I don’t know if it is outside or inside of the scope of JDBC.

Jan Winkler17:11:37

I'm grossly oversimplifying here, but JDBC is a lib that solves a problem of sending a string (a sql query) to a backend and parsing tabular data in the response. There is no support for select, delete or any particular query type, just query in, table out. There is no support for a particular backend implementation either. The support for a particular database is brought in via a driver. If driver provider is nice (and postgres people are!), they may include db-specific APIs in the driver as well. If you want to use this db-specific API, you interact directly with the API. There is nothing JDBC can do for you, you've stepped down a level of abstraction.

kwladyka17:11:52

On the other hand next.jdbc provide support for queries, delete etc. You can be right. I have no idea if COPY should be considered as supported in next.jdbc or it is outside of the scope of this library.

seancorfield17:11:23

Completely outside the scope of next.jdbc.

👍 1
seancorfield17:11:01

The PG docs indicate you can COPY from a file into a table. That is "just" a SQL statement and would work with next.jdbc (because, as Jan says, next.jdbc runs any SQL statement you give it).

Jan Winkler17:11:40

Anyway, what holds you back from using the CopyManager? It seems quite straightforward...

kwladyka17:11:59

> The PG docs indicate you can COPY from a file into a table. Yes, but it will not work for my use case, because I have to modify the file. Reading it and writing to new file to read it again to COPY will be not as efficient.

seancorfield17:11:10

But copying from stdin requires PG-specific stuff -- which that clj-pgcopy library uses.

seancorfield17:11:10

Clojure libraries are often "done" and don't need a steady stream of commits. Why don't you use that library, and if you have problems, ask the maintainer?

kwladyka17:11:32

> Anyway, what holds you back from using the CopyManager? Nothing. It is first time when I will use COPY, so let’s call it process of learning to find the right one library and code.

kwladyka17:11:04

Maybe I will. I don’t know yet.

Jan Winkler17:11:09

(jdbc/with-transaction [tx (jdbc/get-connection datasource)]
  (doto
    (CopyManager. tx)
    (.copyIn "copy mytable from stdin" (io/reader "/path/to/mytable-data"))))

👍 2
seancorfield17:11:04

Also, in future @U0WL6FA77 can you use threads to add detail to an initial post instead of spewing a whole bunch of posts into the main #C1Q164V29 channel? Thanks.

👍 1
kwladyka17:11:12

it will be not from a file, because I have to change the file on the fly.

kwladyka17:11:18

sure, sorry