Fork me on GitHub
#clojure
<
2021-10-02
>
Timofey Sitnikov12:10:50

Good morning All. I am trying to learn how to test the interactions with the database. I have been trying to use https://github.com/Bigsy/pg-embedded-clj, For migrations, I use https://github.com/weavejester/ragtime. How can I copy schema from development db, to the pg-embedded db. I can use pg_dump to export the schema and I simply tried to jdbc/execute! the schema like so:

(log/info (ojdbc/execute! ds (str/split 
                                (slurp "resources/test_db.sql") 
                                #"(?m)\n\s*--;;\s*\n"))) 
But I get this result: Execution error (PSQLException) at org.postgresql.jdbc.PgStatement/checkNoResultUpdate (PgStatement.java:269)

p-himik12:10:08

Does the error text say "A result was returned when none was expected." anywhere?

p-himik12:10:23

You probably have something like

SELECT pg_catalog.set_config('search_path', '', false);
in the result of pg_dump. So you need to turn it and any other potential SELECT statements into statements that don't return anything. Like this:
DO
$$
    BEGIN
        PERFORM pg_catalog.set_config('search_path', '', FALSE);
    END
$$;

1
Timofey Sitnikov12:10:47

OK, I commented it out, it seems to work.

Timofey Sitnikov12:10:07

@U2FRKM4TW do you build schema fixture this way or is there a cleaner way of doin it, is the best approach or should I do this by migrating schema?

p-himik12:10:48

I'm not doing any tests that involve DB right now. But I think it's a reasonable approach when the test DB is created in exactly the same way as you would create a production DB.

seancorfield20:10:04

We do all our production updates via migrations, the same as we do in dev/test and in QA -- with the exception that we have some dev/test-only migrations that do not run on QA or production (these are often setting up fake local data for testing new features). Mostly these are straight-up SQL migrations: we have five migrations that completely teardown databases and rebuild the bare-bones stuff (that only runs on dev/test!) and then we have close to 900 migrations that run to build the dev/test DB up from that bare-bones baseline. The migration process runs all migrations that need to be applied, but skips the dev/test ones on QA and production -- but it's the same process run on all tiers.

seancorfield20:10:15

That migration process is an uberjar we build in CI and auto-deploy to QA (and, later, to production) so the SQL migrations are fully-automated.

Timofey Sitnikov23:10:27

Thank you @U04V70XH6, this makes sense to me. I just had no idea how others are doing it. I will attempt this method.

Timofey Sitnikov00:10:59

OK both methods work fine. The challenge with the pg_dump is the need to edit the exported schema. So I am going with something like:

; ------------------------------------------------------------------------------
(def ds {:dbtype      "postgresql"
         :classname   "org.postgresql.Driver"
         :subprotocol "postgresql"
         :subname     (str "//localhost:" 54321 "/postgres")
         :user        "postgres"})

; ------------------------------------------------------------------------------
(def ragtime-config
  {:datastore  (rjdbc/sql-database ds)
   :migrations (rjdbc/load-resources "migrations")})

; ------------------------------------------------------------------------------
(defn with-pg-fn
  "Startup with the specified configuration;
  executes the function then shuts down."
  [f]
  {:pre [(fn? f)]}
  (try
    (pg/init-pg {:port 54321 :log-redirect "wibble.log"})
    (log/info (repl/migrate ragtime-config))
    (f)
    (finally
      (pg/halt-pg!))))

Timofey Sitnikov01:10:35

OK, amazing day, I had no idea how to test against db this morning, now I able to run tests against my migrated schema, thank you @U2FRKM4TW and @U04V70XH6!

1
👍 1
jjttjj16:10:00

The ordering of keys and vals isn't always consistent right? So this could be potentially buggy:

(defn fuse [kvs]
  (post-complete (apply juxt (vals kvs))
                 (fn [acc]
                   (zipmap (keys kvs) acc))))
(assuming kvs is a map)

noisesmith15:10:54

for a given map, keys and vals always share the same order

noisesmith15:10:04

(in the same clojure version at least)

respatialized16:10:47

if you need to pair them it's best to call (seq m)

p-himik16:10:00

The order is guaranteed to be consistent as long as the map is the same object. The docstrings mention it.

💯 1
jjttjj16:10:05

Oh yeah, should have checked that, thanks!

polymeris20:10:24

This seems a bit strange on first glance :face_with_raised_eyebrow: ... Am I missing something?

(= [:kw] '(:kw))
=> true
(= ['sym] '('sym))
=> false

phronmophobic20:10:19

I think this should help illuminate the issue:

> (pr-str '('sym))
"((quote sym))"
also:
> (= (list (list 'quote 'sym)) '('sym))
true

🤯 1
polymeris20:10:56

Aha (= ['sym] (list 'sym)) ; => true

phronmophobic20:10:34

or

> (= ['sym] '(sym)) 
true
> (= '[sym] '(sym)) 
true
> (= [(quote sym)] (quote (sym))) 
true