Fork me on GitHub
#duct
<
2019-02-07
>
sgerguri12:02:03

How can I set a dev-only JDBC database url when using the duct sql module? My dev.edn looks as follows:

{:duct.database.sql/hikaricp
 {:jdbc-database-url "jdbc:}}
My base config.edn includes the following:
:duct.module/sql {}
When I switch to dev in REPL and run (go), however, I see the following:
Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter
I will take any suggestions on how to resolve this - I do not want to simply slap the url directly into the module config because that URL is dynamic across modules.

sgerguri13:02:54

So it seems that only the base profile key is loaded, hence the dev profile is not being expanded at all.

sgerguri13:02:29

I don't know how to resolve this. I followed the documentation steps to the letter but it does not work for me. Are the docs out of date?

sgerguri13:02:29

(doto (read-config) ig/load-namespaces)
=>
#:duct.profile{:base {:duct.profile/prod {},
                      :duct.profile/dev {:duct.core/environment :development,
                                         :duct.database.sql/hikaricp {:jdbc-database-url "jdbc:}},
                      :duct.profile/local {},
                      :duct.module/logging {},
                      :duct.module/sql {},
                      :duct.module.web/api {},
                      :duct.core/project-ns person-ld-api}}

jeeq13:02:31

@sgerguri I’m not sure if it helps you but I was able to fix the connection issue by writing dev.edn like : `{:duct.database/sql {:connection-uri “jdbc:<postgresql://localhost/db_dev>” :username “postgres_user” :password “db_pw”} } `

sgerguri13:02:18

But if I try to build it, I get basically the same thing back:

(duct/build-config (doto (read-config) ig/load-namespaces) [:duct.profile/dev])
=>
{:duct.profile/prod {},
 :duct.profile/dev {:duct.core/environment :development,
                    :duct.database.sql/hikaricp {:jdbc-database-url "jdbc:}},
 :duct.profile/local {},
 :duct.module/logging {},
 :duct.module/sql {},
 :duct.module.web/api {},
 :duct.core/project-ns person-ld-api}

sgerguri13:02:21

My base config.edn looks as follows:

{:duct.profile/base
 {:duct.core/project-ns person-ld-api

  :duct.profile/dev     #duct/include "dev"
  :duct.profile/local   #duct/include "local"
  :duct.profile/prod    {}

  :duct.module/logging  {}
  :duct.module.web/api  {}
  :duct.module/sql      {}
  }
 }

sgerguri14:02:22

A measure of progress - this seems to work just fine on another duct project that I just created. The difference between these two repos is probably just a week and a half, tops. I checked the configs, versions of dependencies etc. and I don't see any effective difference, yet the earlier repo simply fails to properly initialise the configuration, while the latter one succeeds. Any hints on what I could be missing?

jahson15:02:21

:jdbc-database-url => :jdbc-url?

sgerguri15:02:23

Right, something that is not showing in this snippet is that I am using migrations, specifically:

:duct.migrator/ragtime
 {:migrations [#ig/ref :person-ld-api/create-database
               #ig/ref :person-ld-api/create-table]}

 [:duct.migrator.ragtime/sql :person-ld-api/create-database]
 #duct/include "migrations/001-create-db.edn"

 [:duct.migrator.ragtime/sql :person-ld-api/create-table]
 #duct/include "migrations/002-create-table.edn"
The above is in my config.edn and is what breaks it, basically.

sgerguri15:02:52

I guess I am using #duct/include incorrectly - I want to simply refer to another EDN file and have it be included in here.

sgerguri15:02:13

True, @jahson - I have since fixed that, however, and the migrations are what break it.

sgerguri15:02:25

It seems that if one uses duct with ragtime one is effectively forced out of the edn migration files - either everything ends up being included in the config.edn file, or one has to make sql up/down files and refer to those. Neither feels like a particularly great solution.

jahson15:02:33

:duct.migrator/ragtime
 {:database #ig/ref :xxx.database/master
  :migrations-table "xxx_migrations"
  :migrations [#ig/ref :xxx.migration/create-table1]}

[:duct.migrator.ragtime/sql :xxx.migration/create-table1]
{:up ["CREATE TABLE table1;
 :down ["DROP TABLE table1"]}
Are these migrations look like this?

sgerguri15:02:28

The files I am referring to via the #duct/include have basically

{:up ["CREATE TABLE table1;"]
 :down ["DROP TABLE table1;"]}
in them.

sgerguri15:02:52

I just thought I could make config.edn more concise by simply referring to these through that include.

jahson15:02:50

I think there is nothing wrong with #duct/include

jahson15:02:13

What is happening? Exceptions?

sgerguri15:02:24

(prep)
Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter

sgerguri15:02:26

ExceptionInfo Error on key :duct.migrator/ragtime when building system 
Caused by:
IllegalArgumentException db-spec null is missing a required parameter
	clojure.java.jdbc/get-connection (jdbc.clj:292)
	clojure.java.jdbc/get-connection (jdbc.clj:176)
	ragtime.jdbc/get-table-metadata (jdbc.clj:27)
	ragtime.jdbc/get-table-metadata (jdbc.clj:24)
	ragtime.jdbc/table-exists? (jdbc.clj:38)
	ragtime.jdbc/table-exists? (jdbc.clj:36)
	ragtime.jdbc/ensure-migrations-table-exists (jdbc.clj:41)
	ragtime.jdbc/ensure-migrations-table-exists (jdbc.clj:40)
	ragtime.jdbc.SqlDatabase (jdbc.clj:61)
	ragtime.core/migrate-all (core.clj:57)
	ragtime.core/migrate-all (core.clj:33)
	duct.migrator.ragtime/migrate (ragtime.clj:76)

jahson15:02:07

Interesting. I thik the issue is here https://github.com/duct-framework/migrator.ragtime/blob/master/src/duct/migrator/ragtime.clj#L63 It is happening because you should explicitly set :database I think.

jahson15:02:04

{:duct.migrator/ragtime
 {:database   #ig/ref :duct.database/sql
  :logger     #ig/ref :duct/logger
  :strategy   :rebase
  :migrations [#ig/ref :foo.migration/create-foo-table]}}
Looks like you could just copy this from README.

sgerguri15:02:06

I'm not sure that's it - I have database settings in my dev.edn.

sgerguri15:02:48

My dev.edn looks as follows:

{:duct.database/sql
 {:connection-uri "jdbc:"
  :username "PLACEHOLDER"
  :password "PLACEHOLDER"}}

jahson15:02:54

You should provide :database for :duct.migrator/ragtime

sgerguri15:02:10

Yes - and I am through the dev profile.

sgerguri15:02:28

That section is demoted in the module, so it gets overriden if there is a value for it.

jahson15:02:14

Can you try to use an explicit :database to check if it will work?

sgerguri15:02:35

I can - will update here later after I've tried.

jahson15:02:03

Sorry if I sound a little bit rude — I not a native nor fluent english speaker.

sgerguri15:02:17

Not at all - and I hope I did not come across that way either. I am just trying it out.

weavejester15:02:49

Hi, can I help in any way?

sgerguri15:02:57

Ah, the man himself!

sgerguri15:02:07

Yes please, let me summarise.

sgerguri15:02:32

Basically, I set up a duct project with DB and routes, and was struggling to get the DB layer set up.

sgerguri15:02:58

I set the project up again and am seeing success in instantiating hikari but the moment I specify some DB migrations it all blows up.

weavejester15:02:01

Could you post your config file, and the exception that occurs?

sgerguri15:02:53

Contents of config.edn:

{:duct.profile/base
 {:duct.core/project-ns ld-api

  :duct.router/cascading
  []}

 :duct.profile/dev   #duct/include "dev"
 :duct.profile/local #duct/include "local"
 :duct.profile/prod  {}

 :duct.module/logging {}
 :duct.module.web/api {}
 :duct.module/sql {}

 ;; == DB migrations ==

 :duct.migrator/ragtime
 {:migrations [#ig/ref :ld-api/create-database
               #ig/ref :ld-api/create-table]}

 [:duct.migrator.ragtime/sql :ld-api/create-database]
 #duct/include "migrations/001-create-db.edn"

 [:duct.migrator.ragtime/sql :ld-api/create-table]
 #duct/include "migrations/002-create-table.edn"
 }

sgerguri15:02:12

Contents of dev.edn:

{:duct.database/sql
 {:jdbc-url "jdbc:"
  :username "PLACEHOLDER"
  :password "PLACEHOLDER"}}

sgerguri15:02:28

Error:

(prep)
Execution error (IllegalArgumentException) at clojure.java.jdbc/get-connection (jdbc.clj:292).
db-spec null is missing a required parameter

sgerguri15:02:51

The above happens when calling (prep) after calling (dev).

weavejester15:02:34

Hm. What happens if you change :duct.database/sql to :duct.database.sql/hikaricp in your dev.edn?

weavejester15:02:34

Well at least that's consistent 🙂

weavejester15:02:01

Oh, I see a problem.

sgerguri15:02:05

True, though that provides little comfort to me. 😄

weavejester15:02:24

Your migrator keys are outside the base profile.

weavejester15:02:06

You should dump them under :duct.profile/base, or create a new profile for them.

weavejester16:02:04

It's an easy mistake to make, because in older versions that would have worked. But there were too many issues with putting modules and non-modules in the same configuration.

weavejester16:02:16

So non-module (component) keys need to be under a profile.

weavejester16:02:30

Which is just a module that merges its contents into the configuration.

sgerguri16:02:17

Unfortunately, now the config isn't being expanded properly.

sgerguri16:02:12

(pprint config)
{[:duct.migrator.ragtime/sql :ld-api/create-person-table] {:up ["CREATE TABLE person (...);"],
                                                           :down ["DROP TABLE person;"]},
 :duct.profile/prod {:duct.core/requires #ig/refset :duct.profile/base,
                     :duct.core/environment :production},
 :duct.profile/dev {:duct.database.sql/hikaricp {:jdbc-url "jdbc:",
                                                 :username "PLACEHOLDER",
                                                 :password "PLACEHOLDER"},
                    :duct.core/requires #ig/refset :duct.profile/base,
                    :duct.core/environment :development},
 :duct.migrator/ragtime {:migrations [#ig/ref :ld-api/create-database
                                      #ig/ref :ld-api/create-person-table]},
 :duct.router/cascading [],
 :duct.profile/local {:duct.core/requires #ig/refset :duct.profile/base},
 :duct.module/logging {:duct.core/requires #ig/refset :duct/profile},
 :duct.module/sql {:duct.core/requires #ig/refset :duct/profile},
 [:duct.migrator.ragtime/sql :ld-api/create-database] {:up ["CREATE DATABASE ld;"],
                                                       :down ["DROP DATABASE ld;"]},
 :duct.module.web/api {:duct.core/requires #ig/refset :duct/profile},
 :duct.core/project-ns ld-api}

weavejester16:02:24

That looks pretty mixed up... sorry, can you post your config.edn again?

sgerguri16:02:57

{:duct.profile/base
 {:duct.core/project-ns ld-api

  :duct.router/cascading
                        []

  :duct.profile/dev     #duct/include "dev"
  :duct.profile/local   #duct/include "local"
  :duct.profile/prod    {}

  :duct.module/logging  {}
  :duct.module.web/api  {}
  :duct.module/sql      {}

  ;; == DB migrations ==

  :duct.migrator/ragtime
                        {:migrations [#ig/ref :ld-api/create-database
                                      #ig/ref :ld-api/create-person-table]}

  [:duct.migrator.ragtime/sql :ld-api/create-database]
                        #duct/include "migrations/001-create-db.edn"

  [:duct.migrator.ragtime/sql :ld-api/create-person-table]
                        #duct/include "migrations/002-create-person-table.edn"}
 }

weavejester16:02:42

Right. So you need to put component keys in profiles, module keys outside.

weavejester16:02:46

So:

{:duct.profile/base
 {:duct.core/project-ns ld-api
  :duct.router/cascading []

  :duct.migrator/ragtime
  {:migrations [#ig/ref :ld-api/create-database
                #ig/ref :ld-api/create-person-table]}

  [:duct.migrator.ragtime/sql :ld-api/create-database]
  #duct/include "migrations/001-create-db.edn"

  [:duct.migrator.ragtime/sql :ld-api/create-person-table]
  #duct/include "migrations/002-create-person-table.edn"}

 :duct.profile/dev     #duct/include "dev"
 :duct.profile/local   #duct/include "local"
 :duct.profile/prod    {}

 :duct.module/logging  {}
 :duct.module.web/api  {}
 :duct.module/sql      {}}

sgerguri16:02:05

Ah, should've paid closer attention to https://github.com/duct-framework/docs/blob/master/GUIDE.rst. I went by what the READMEs and Wiki of duct framework and modules said, and mashed it up together based on that.

weavejester16:02:31

I'm working on some better docs that should make things clearer

sgerguri16:02:48

Happy to help out once I've wrapped my head around stuff comfortably. So I have a question.

sgerguri16:02:01

What if I want to override some component-level stuff from a non-base profile?

sgerguri16:02:17

Technically, the module keys are still part of the base profile.

sgerguri16:02:41

So does that mean that components can go into any profile (nested in base) and it will all work, right?

weavejester16:02:28

Can you give an example of what you mean?

sgerguri16:02:47

For example, could my dev.edn file look like the following?

{:duct.database.sql/hikaricp
 {:jdbc-url "jdbc:"
  :username "postgres"
  :password "postgres"}

:duct.migrator/ragtime
{:migrations [...]}

...
}

weavejester16:02:11

Yes, that's right.

sgerguri16:02:45

Perfect, that makes sense now.

weavejester16:02:01

The only things that currently can't be put in profiles are modules.

sgerguri16:02:13

Thanks a lot, James. I might try my hand at a few PRs that hopefully clear up how things should be used now that I have a better understanding of it.

weavejester16:02:34

No problem. Let me know if you have any more questions.