Fork me on GitHub
#duct
<
2020-09-08
>
Danny Almeida06:09:32

Does anyone know if the keys from local.edn and dev.edn are merged into config.edn before initialising modules ? I'm using https://github.com/hden/duct.module.datomic and my config.edn looks like this:

:duct.module/datomic
 {
  :server-type :peer-server,
  :secret #duct/env "DATOMIC_DB_PASS",
  :access-key #duct/env "DATOMIC_DB_ACCESS_KEY",
  :endpoint #duct/env "DATOMIC_HOST",
  :validate-hostnames false
  :database #duct/env "DATOMIC_DB_NAME"
  }
 }
And my local.edn has the connection values defined under the same key, but for some reason, the config passed to the module during initialisation, does not contain the values from local.edn and since I don't define the environment variables in dev profile, the config passed has nil values for all keys except :server-type. The module just preps :duct.database/datomic and :duct.migrator.ragtime/datomic integrant keys.

iarenaza09:09:27

@dionysius.almeida As long as you have the corresponding :duct.profile/dev #duct/include "dev" and :duct.profile/local #duct/include "local" in your base profile definition, yes, they are merged into config.edn before initialising the modules.

Danny Almeida12:09:04

@larenaza both includes are present in the base definition. After running (prep), I can see that the values are indeed replaced with that of local.edn file, however :duct.database/datomic key is still being initialised with nil values. :duct.database/datomic {:server-type :peer-server, :secret nil, :access-key nil, :endpoint nil, :validate-hostnames false, :database nil, :duct.core/requires #{#function[duct.core/eval10494/fn--10495/fn--10496] #function[duct.core/eval10494/fn--10495/fn--10496] #function[duct.core/eval10494/fn--10495/fn--10496]}}

iarenaza17:09:14

@dionysius.almeida I'm a bit lost. Your Duct module is merging the configuration you pass it as arguments (what you call [`options` here](https://github.com/hden/duct.module.datomic/blob/master/src/duct/module/datomic.clj#L15)) . If what you pass to the module isn't defined (because the arguments you are passing try to get their values from env variables that are not defined), why do you expect your module to produce a different configuration? Specially since the merge of those values doesn't specify any merging "preeminence" (e.g, displace, replace, demote, etc).

Danny Almeida07:09:26

Sorry for the confusion. This is not my module, I'm just using it to learn duct + datomic integration. Although the main config.edn has these variable being pulled from enviroment variables for release, my local dev setup has them defined in dev/resources/local.edn file. But for some reason, they are not being merged with the config.edn file and so during module initialisation, the module get a config with nil values for everything except for :server-type.

iarenaza07:09:16

Can you share your full local.edn and config.edn files?

Danny Almeida12:09:47

Sure...here's my config.edn, local.edn and output after running (prep) command. Thank you for helping me out 🙂

iarenaza14:09:30

@dionysius.almeida Ok, i think I know what your problem is. If you have a look at https://github.com/duct-framework/duct/wiki/Configuration#duct-base-profiles-and-modules you'll see that there are three parts to prep the configuration. Duct profiles (like :duct.profile/local, the one that deals with local.edn) are merged in the base config. And the above link states that "The base config is a map of data inside of the :duct.profile/base key". So Duct reads config.edn, takes the data from the :duct.profile/base key and creates the base config. Next it takes all the profiles keys from config.edn and merges the associated data with the base config. That's why you see there is a :duct.module/datomic key in your config var (the one you show in the repl-output.txt file). And it has the exact same values that you configured in local.edn. Finally it takes all the modules keys from config.edn and runs the configuration through them. But the module key that you are trying to "inject" from local.edn is not in config.edn! It's only in the configuration merged so far, but not in config.edn. So Duct doesn't know about it and doesn't try to run the configuration through it. The only Datomic module key it knows about it the one in config.edn. And that one uses the settings from environment variables. And as you said, they are not defined, so they end up being nil. One (simplified, but still mostly true) way to see it is that Duct modules need to live outside the :duct.profile/base and :duct.profile/* keys for them to be processed. In this particular case I don't see and easy way around it, as the Datomic module expects the configuration settings to be passed in as the configuration map for its Integrant key. And there is no way (that I know of) to specify the configuration settings in the base profile (so we can override them from dev.edn or local.edn) and ig/ref to their key from the Datomic module Integrant key. But if you modified the Datomic module to look for the configuration settings in the config map instead of passing them as the options argument to the init-key multimethod, you could achieve what you want. If you wanted to make the modification a bit more generic, instead of hardcoding the key to look for in the config map, you could pass that as a configuration option to the init-key multimethod. Something along these lines:

(ns duct.module.datomic
  (:require [duct.core :as core]
            [integrant.core :as ig]))

(defn- get-environment [config options]
  (:environment options (:duct.core/environment config :production)))

(def ^:private logger-configs
  {:production
   {:duct.logger.timbre/cast {}
    :duct.logger/timbre
    ^:demote {:appenders ^:displace {:duct.logger.timbre/cast (ig/ref :duct.logger.timbre/cast)}}}})

(defmethod ig/init-key :duct.module/datomic [_ options]
  (fn [config]
    (core/merge-configs
     config
     {:duct.database/datomic (ig/ref (:settings-key options))}
     {:duct.migrator.ragtime/datomic
      ^:demote {:database   (ig/ref :duct.database/datomic)
                :logger     (ig/ref :duct/logger)
                :migrations []}}
     (logger-configs (get-environment config options)))))
coupled with somethig like this in config.edn (using :duct/const, available from duct.core 0.7.0 onwards):

iarenaza14:09:30

{:duct.profile/base

 {:duct.core/project-ns duct-redis-example

  :duct.handler/root
  {:router #ig/ref :duct.router/reitit
   ;; use this for applying middleware to entire root handler
   ;; :middleware [#ig/ref :duct-redis-example.handler.middleware/wrap-admin]
   }
  :duct.router/reitit
  {:routes
   ["/"
    ["example" {:handler #ig/ref :duct-redis-example.handler/example}]
    ["datomic-example" {:handler #ig/ref :duct-redis-example.handler/datomic-example}]
    ["admin" {:handler #ig/ref :duct-redis-example.handler/example
              :middleware [:wrap-admin ]}]
    ]
   :reitit.ring/opts
   {:reitit.middleware/registry
    {:wrap-admin #ig/ref :duct-redis-example.handler.middleware/wrap-admin}}
   }
  :duct-redis-example.handler.middleware/wrap-admin {}
  :duct-redis-example.handler/example {:db #ig/ref :duct.database.redis/carmine}
  :duct-redis-example.handler/datomic-example {:db #ig/ref :duct.database/datomic }

  :duct.server.http/jetty {:port #duct/env ["SERVER_PORT" Int]}

  :duct.database.redis/carmine
  {:spec
   {:host #duct/env "REDIS_HOST",
    :port #duct/env ["REDIS_PORT" Int]}
   }

  [:duct/const :datomic/settings]
  {
   :server-type :peer-server,
   :secret #duct/env "DATOMIC_DB_PASS",
   :access-key #duct/env "DATOMIC_DB_ACCESS_KEY",
   :endpoint #duct/env "DATOMIC_HOST",
   :validate-hostnames false
   :database #duct/env "DATOMIC_DB_NAME"
   }
 }

 :duct.profile/dev   #duct/include "dev"
 :duct.profile/local #duct/include "local"
 :duct.profile/prod  {}

 :duct.module/logging {}
 :duct.module.web/api {}
 :duct.module/datomic {:settings-key :datomic/settings}
 }
and this in local.edn:
{[:duct/const :datomic/settings]
 {
  :server-type :client-server-local-value,
  :secret :secret-local-value
  :access-key :access-key-local-value
  :endpoint :endpoint-local-value
  :validate-hostnames :validate-hostnames-local-value
  :database :database-local-value
  }
 }

Danny Almeida01:09:02

Thank you for the detailed explanation. I did figure out that the way modules are initialised is different from other keys, but this explanation now help me understand why. Since, this is not my module, I don't think i would be able to change it. I also found a way to deal this with....instead of adding my configuration in :duct.module/datomic, I define the configuration in :duct.database/datomic key in base profile and then override it in local.edn file using this same key. But your solution is something I will keep in mind as I'm trying to write a module to use datomic pro which uses a URI string, instead of a map to connect to datomic transactor and not peer server. This module uses client API library and not peer library api, so i might put toghether something which works for peer library. Thank you again for your time and have a nice weekend 🙂