This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-01-19
Channels
- # adventofcode (4)
- # beginners (80)
- # boot (4)
- # cbus (2)
- # cider (62)
- # clara (18)
- # cljs-dev (8)
- # cljsrn (10)
- # clojure (139)
- # clojure-brasil (3)
- # clojure-dev (27)
- # clojure-italy (1)
- # clojure-russia (3)
- # clojure-spec (4)
- # clojure-uk (47)
- # clojurescript (102)
- # core-async (10)
- # cursive (7)
- # datomic (71)
- # emacs (32)
- # fulcro (99)
- # funcool (1)
- # hoplon (3)
- # jobs (1)
- # jobs-discuss (6)
- # jobs_rus (2)
- # leiningen (3)
- # luminus (2)
- # lumo (14)
- # mount (7)
- # off-topic (19)
- # re-frame (25)
- # ring-swagger (4)
- # rum (3)
- # shadow-cljs (142)
- # specter (2)
- # sql (16)
- # timbre (1)
- # vim (3)
Note that both create-table-ddl
and drop-table-ddl
can be passed an options hash map as the last argument and, if your database supports it, you can say {:conditional? true}
and the generated SQL will contain an EXISTS
test (producing DROP TABLE IF EXISTS fruit
and CREATE TABLE IF NOT EXISTS fruit ...
). ^ @jisaacs46
Wow awesome thanks man that helps a ton. @seancorfield
@jisaacs46 Are these two databases within the same DB instance? i.e., if your :user
has access to both databases/schemas then you can do it in one SQL statement (with execute!
).
Or, if the data you're moving can fit in memory, just query
it from one DB into a local binding and then insert!
it into the other database -- after performing whatever in-memory transformations you need to apply.
Ah, I answered before I expanded your code -- so I was being speculative...
I can't think of a way to do a single update
based on data in-memory... but my comment about a user with privileges on both DBs would be my first optimization -- if that is possible.
Ok great I’ll try something like that I haven’t been able to get the execute! to work yet. But I’ll give it another try and see if I can get something going. Thank you @seancorfield you have been really helpful. I have a Clojure project connecting to 3 databases and pulling all the data into one database so that I can compare the tables. Clojure and this slack channel has been a life saver thanks again.
Feel free to post code and stacktraces or whatever. Happy to help.
@jisaacs46 We do a lot of general database analysis and maintenance via a REPL -- a Socket REPL connected into one of our production processes 🙂
I am wondering if there is something i could do to speed up getting the data from multiple databases using my above snippet. It takes about 3-5 minutes to run. would execute! speed this up?
Is there a plug-in for Clojure where I can download data from a database into a file in the form of insert statements and or json format?
spit
, slurp
, jdbc/select
and jdbc/insert!
or jdbc/insert-multi!
probably get you where you want to go, possibly with judicious use of partition-all
Also is there a way to copy out millions of rows quickly into a file in clojure I use datagrip and it can copy data to a file really fast I’m wondering if there is something in Clojure that can copy data out quickly.
If you’re talking millions of rows, you’ll probably want to open a writer yourself, make sure you don’t retain the head of your results seq, and do whatever annoying things you have to do with your db connection to ensure it isn’t buffering the entire resultset.
Writing JVM-based tools to replicate what existing specialist SQL tools you already have is pretty much always going to produce something slower.