This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-11-06
Channels
- # announcements (3)
- # babashka (1)
- # beginners (26)
- # calva (1)
- # cider (17)
- # clj-commons (16)
- # clj-kondo (11)
- # clojure (21)
- # clojure-europe (9)
- # clojure-norway (1)
- # clojure-portugal (2)
- # clojure-spec (8)
- # clojure-uk (4)
- # clojurescript (35)
- # datomic (5)
- # emacs (9)
- # figwheel-main (15)
- # fulcro (26)
- # honeysql (1)
- # lsp (5)
- # off-topic (2)
- # polylith (1)
- # rdf (6)
- # re-frame (4)
- # reagent (15)
- # reitit (9)
- # releases (2)
- # shadow-cljs (4)
- # sql (25)
- # squint (2)
- # xtdb (7)
Someone have ready solution with next.jdbc for postgresql and COPY
with STDIN
? Reading CSV, remove columns, write to STDIN. As I understand binary
is the most performant way.
https://www.npgsql.org/doc/copy.html https://stackoverflow.com/questions/14242117/java-library-to-write-binary-format-for-postgres-copy I found this example. I am not sure if it is possible in next.jdbc just like that. It looks like it needs special support. Can I use at least part of the next.jdbc code or there is really nothing to use in next.jdbc for COPY?
I know there is https://github.com/jgdavey/clj-pgcopy but last change was 2 years ago and I would like to use 1 library instead of 2 for postgresql.
(org.postgresql.copy CopyManager
CopyIn
PGCopyOutputStream)
Perhaps it needs this ^
But I am still figuring this out possibilitieshmm I think it is a little harder, because with binary
it also need data type support
But COPY
is postgres-specific, there is nothing to be done on JDBC level of abstraction and definitely not in next-jdbc, which is "just" a JDBC wrapper... You found the copy API in psql driver and that is what you should use. Either directly via java interop or using clj-pgcopy, if it works with your/current psql driver.
I'm grossly oversimplifying here, but JDBC is a lib that solves a problem of sending a string (a sql query) to a backend and parsing tabular data in the response. There is no support for select
, delete
or any particular query type, just query in, table out. There is no support for a particular backend implementation either.
The support for a particular database is brought in via a driver. If driver provider is nice (and postgres people are!), they may include db-specific APIs in the driver as well. If you want to use this db-specific API, you interact directly with the API. There is nothing JDBC can do for you, you've stepped down a level of abstraction.
On the other hand next.jdbc provide support for queries, delete etc. You can be right. I have no idea if COPY
should be considered as supported in next.jdbc
or it is outside of the scope of this library.
The PG docs indicate you can COPY
from a file into a table. That is "just" a SQL statement and would work with next.jdbc
(because, as Jan says, next.jdbc
runs any SQL statement you give it).
Anyway, what holds you back from using the CopyManager? It seems quite straightforward...
> The PG docs indicate you can COPY from a file into a table. Yes, but it will not work for my use case, because I have to modify the file. Reading it and writing to new file to read it again to COPY will be not as efficient.
But copying from stdin requires PG-specific stuff -- which that clj-pgcopy library uses.
Clojure libraries are often "done" and don't need a steady stream of commits. Why don't you use that library, and if you have problems, ask the maintainer?
> Anyway, what holds you back from using the CopyManager? Nothing. It is first time when I will use COPY, so let’s call it process of learning to find the right one library and code.
Do it. 🙂
(jdbc/with-transaction [tx (jdbc/get-connection datasource)]
(doto
(CopyManager. tx)
(.copyIn "copy mytable from stdin" (io/reader "/path/to/mytable-data"))))
Also, in future @U0WL6FA77 can you use threads to add detail to an initial post instead of spewing a whole bunch of posts into the main #C1Q164V29 channel? Thanks.