Fork me on GitHub
#duct
<
2018-11-07
>
lambder11:11:16

Hi, How can I use data collected by CLI args with duct config defined in edn file? With duct config as a data-structure is easy as I can inject the values into the map but I don't know how to do it if the config is sourced from edn.

weavejester11:11:36

@lambder You can either update your -main method to update the configuration before it's prepped, or - preferably - use environment variables instead of CLI args and take advantage of the #duct/env reader tag.

weavejester12:11:59

@lambder Take a look at the generated -main function. The configuration is read in, then prepped and executed, but after it's read in and before it's prepped you can modify it as you want.

weavejester12:11:20

I don't know why you'd want to set variables during runtime. That seems a bad idea.

lambder12:11:24

so I use org.clojure/tools.cli before I construct duct config

lambder12:11:49

I want to some of the cli args to be injected to duct

lambder12:11:03

it's too late to set the env vars

lambder12:11:12

I could set the java props tho

weavejester12:11:29

Yes, if you really want to do that. Are you using 0.10 or 0.11-beta?

lambder12:11:20

[duct/core "0.6.2"]

weavejester12:11:13

@lambder So your -main function should look like:

(defn -main [& args]
  (let [keys (or (duct/parse-keys args) [:duct/daemon])]
    (-> (duct/read-config (io/resource "bar/config.edn"))
        (duct/prep keys)
        (duct/exec keys))))

weavejester12:11:47

You can alter the configuration with whatever function you want after read-config

weavejester12:11:16

Though I'd advise sticking to environment variables.

lambder12:11:10

the keys are the list of ig keys . correct?

lambder12:11:28

what rather than selecting the keys I want to take a http port form cli?

lambder12:11:39

I know I can use env var for this.

lambder12:11:54

I'm using this as an illustration of the case only.

lambder12:11:09

so, how can I inject http port form cli to duct?

weavejester12:11:11

It's up to you. You can pass it as a raw argument or you can define a flag to take it, like -p 3000.

lambder12:11:15

but how to pass it to duct conf?

weavejester12:11:30

The Duct configuration is just a map, returned by read-config

weavejester12:11:37

So you can use assoc-in as normal

weavejester12:11:40

Or whatever you want.

lambder13:11:41

Ok, I've managed to implement what I wanted. I'm sharing here just in case you want to include it in the duct framework.

(defmacro local 
  "A string reader providing values of local bindings"
  []
  (let [lvals (vec (keys &env))
        lvars (mapv keyword (keys &env))]
    `(let [local-vars-map# (into {}
                     (map vector
                          ~lvars
                          ~lvals))]
       (fn [v#]
         ((keyword v#) local-vars-map#)))))
Usage:
(let [a 123
      b "xyz"]
  (duct/read-config
    (java.io.StringReader. "{:foo  {:x #ig/val a}
                             :bar  {:x #ig/val b}}")
    {'ig/val (local)}))
produces: {:foo {:x 123}, :bar {:x "xyz"}}

lambder13:11:59

instead of symbols the (local) string reader cold take keywords or both

lambder13:11:11

e.g

(let [a 123
      b "xyz"]
  (duct/read-config
    (java.io.StringReader. "{:foo  {:x #ig/val :a}
                             :bar  {:x #ig/val :b}}")
    {'ig/val (local)}))
works too

lambder14:11:55

@weavejester I use [duct/core "0.6.2"] which seems to be latest. My print reader works with duct/read-config but if my config includes other one by :duct.core/include it looks like my reader is no longer being applied for the included edn. Any idea how to fix it?

weavejester14:11:02

@lambder The readers should be passed straight through to any includes. It might be something weird with the macro you're using, though it produces a closure at the end of it so it should be safe.

weavejester14:11:52

I think this is something you'll need to investigate yourself if you want to head down this route. If it turns out to be a bug in duct/core, send me an issue report and I'll put together a fix.

weavejester14:11:24

I'd also be interested in knowing why you're taking this approach to begin with 🙂

lambder14:11:50

I take this approach to have the full duct configs in edn files. The injected dynamic values (established in runtime before the duct system is initialised) are used in the duct config on many levels (in the sense of nested maps).

weavejester14:11:27

What are the dynamic values for?

lambder14:11:33

Transforming that would require post or pre walking the data structure and determining what should be repalaced.

lambder14:11:48

the dynamic values are coming from the user

lambder14:11:15

imagine the user runs it as a command line providing the dynamic values as args

lambder14:11:38

I could convert the loaded duct conf as you suggested but is a but cumbersome

lambder14:11:53

with my approach (local bindings) it is resolved automatically

lambder14:11:59

I create my conf by:

(let [insert-record        (:insert-record opts)]
                              (duct/read-config
                                (io/resource duct-config ) {'local (u/local)}))

lambder14:11:07

which works

lambder14:11:23

if the duct-config is a string to my prod duct conf

lambder14:11:37

if I point it to my dev one (dev includes prod)

lambder14:11:51

the duct claims there is no #local reader

weavejester14:11:57

Okay, and the 'local isn't being transferred to files included with :duct.core/include?

lambder15:11:01

I just killed my REPL

lambder15:11:17

it still is not working

lambder15:11:52

I've ended up to repeat all the config from prod in dev (except the difference)

weavejester15:11:26

Are you passing the readers to the prep function as well as read-config?

lambder15:11:12

no, just to read-config

lambder15:11:23

should I do with prep as well?

lambder15:11:04

if yes how can I supply the readers and require prep for all the keys, not just selected ones?

weavejester15:11:11

prep is what loads the include files, so you need to supply the readers to that if you want the include files to have readers.

weavejester15:11:58

@lambder You should just need to supply your reader function to both the read-config and prep stage.

weavejester15:11:08

The next version of Duct makes this process simpler.

lambder15:11:59

(defn prep
  "Prep a configuration, ready to be initiated. Key namespaces are loaded,
  resources included, and modules applied.

  Like `init` and other Integrant functions, `prep` can be limited to a subset
  of keys. The keys supplied indicate which keys will be later initiated, and
  therefore which keys to load. Modules are always loaded and applied.

  A map of options may also be supplied. Currently the only supported option is
  `:readers`, which should contain a map of data readers that will be used when
  reading configurations imported through `:duct.core/include`. "
  ([config]
   (prep config nil))
  ([config keys]
   (prep config keys {}))
  ([config keys {:keys [readers] :or {readers {}}}]
   (-> config
       (apply-includes (memoize #(read-config % readers)))
       (doto (ig/load-namespaces [:duct/module]))
       (apply-modules)
       (doto (as-> cfg (if keys
                         (ig/load-namespaces cfg keys)
                         (ig/load-namespaces cfg)))))))

lambder15:11:31

if I call (prep conf :reders my-readers)

lambder15:11:51

which arity will be called?

weavejester15:11:23

You mean: (prep conf {:readers my-readers})?

lambder15:11:49

that would be the 2-arity right?

weavejester15:11:59

Yes, because there are 2 arguments.

lambder15:11:07

which is `([config keys] (prep config keys {}))`

lambder15:11:22

that would break

weavejester15:11:30

Yeah, so just supply the keys, or nil if you want the default.

weavejester15:11:46

Though -main supplies [:duct/daemon] as its default, mind.

lambder15:11:46

ok, that was what I meant how do I require all the keys

weavejester15:11:29

I believe just supply nil, though usually you just want [:duct/daemon] if you're running from -main, and nil if you're running from the REPL.

lambder15:11:45

I run it a bit diferently

lambder15:11:52

I construct a duct system

lambder15:11:22

and then I have my own main driver which picks different duct components from the system

lambder15:11:38

but that is a different story 😉

weavejester15:11:22

It sounds like you're doing some sort of customer/user-specific sharding.

lambder18:11:39

@weavejester thank to your advices everything works perfect now. Thanks!