Fork me on GitHub

Aero can do environment variables, but generally thinks you shouldn't :D


@paulspencerwilliams it's less obvious what your program is doing. Have you ever been confused as to why something as happening only to discover that is because of some environment thing (including files in odd places)


@dominicm I do know what you mean, but it's particularly useful when using containers to have config injected as ENVs and as long as it's documented it's really not that big a risk or mystery


@dominicm I'm just uncomfortable with config in the codebase these days, based on several bad experiences... 😉


@maleghast most of the time you just need a variable to tell you you're in prod. And you can derive everything from there.


@dominicm I'm all for that if it works for you / your team / your codebases etc. I am just not ok with it, currently. I am prepared to be persuaded given more spare head-cycles to consider it...


This "Where do I draw the line between code and data (re: properties files/environment variables)" is one that's perplexed me for over 20 years. Even before I moved to Clojure I had discovered thru using ORM's and frameworks that the 'config' is just as much a part of the behaviour of the system as the 'code'. The distinction is even less apparent in Clojure. In my experience if you change anything except trivial config you probably need to retest the entire app/component/system anyway so I tend to limit the externalisation of 'config' to things I absolutely know will change regularly in production and things that change in the journey to production (i.e. environment specific stuff like URLs and username/passwords). Anything complex I tend to encode in different files per environment (EDN is a good choice for format in Clojure or property files for Java etc.) and then have an environment variable to tell the code which file to load (i.e. an environment environment variable!).


Basically config that changes behaviour is code


So treat it that way: test it, version control it, etc.


This is one area I really can speak form authority having written dozens of systems that were naively config driven and maintained many more. I've paid the price of the assumption that config makes systems easier to adapt than code. It's the other way around as you end up with lots of boiler plate to deal with decisions based on config and you always guess wrong and the thing you didn't configure for changes.


Having said all that phew! you would think I'm anti framework....but I'm not! I think frameworks that use config are fine but the config must coexist happily with the code (not be externalised) and there must be tools and support to allow testing. So I'm interested in Arachne's DSL for config for example.

Rachel Westmacott08:02:23

"I am prepared to be persuaded given more spare head-cycles to consider it..." @maleghast I love this phrase.

Rachel Westmacott08:02:53

I think in the past there was this idea that you could redeploy the same code with different config files - but now people are all like: "let's put all the code and config down the same test-and-deploy-pipeline every time"

Rachel Westmacott08:02:19

I feel like it shouldn't really matter if you have lots of environmental factors that your code depends on - because your environment is also going to be built from code, so its just more code really.... plus some people really like their "12-factor apps"


@maleghast me too. Not sure my argument has convinced you. Sounds like we have experiences at opposite ends of the scale! On a scale of maintainability nightmares->reasonable to maintain, I have worked with a half dozen case tools that generate code (words fail me for how bad these are), then written and used dozens of 'flexible, config driven/rules engine based systems' shudders, lots of systems with environment specific externalised config where everything that could be changed is external, and finally config packaged with code and a few simple environment variables to pick the right config. Much prefer the later.

Rachel Westmacott08:02:52

(having spoken in favour of env vars, I will just add that (obviously?) I think that minimising the surface area between the env and the code is going to be a good idea)


@agile_geek - Need time to digest this; am actually supposed to be working... Get back to you in a bit? (Sorry 😞 And thanks for the long post, I am interested 🙂 )

Rachel Westmacott08:02:54

I like to think that researching coding best practices (ie. chatting on slack) is a key part of my job, and part of what makes me valuable. 😄


config is stuff that changes depending on where you deploy


anything else is just your app


and RE: having loads of env vars, I try and resolve that with useful defaults


@glenjamin I think you just summarised my rant to 3 sentences! simple_smile


i pretty much agree with everything on that 12 factor page


it might not all be perfect advice, but you need a damn good reason to deviate from it IME


I would extend to say anything that has a lot of stuff that changes depending on where you deploy but doesn't change frequently should be packaged in your deployment unit with a simple env variable pointing to it.


as a concrete example, the app i’m currently working on takes an env var called HTTPS_ONLY. If you enable it you get a 301 redirect when accessing over http, HSTS headers and the cookies are all flagged as secure. It’s off by default so dev on localhost is easier


meh, that stuff lives in my configuration management tool generally


I guess that depends on the config and the CM tool but I have no problem with that.


@agile_geek - Right, read it, agree wholeheartedly, but my current situation is in a PHP environment and having the config in an entirely separate Git Repo, by environment (Dev, Stage, Prod) that has (until recent staff cuts) had much narrower access was the only way to stay sane. It is fair to say that the approach you describe, config per environment and then one ENV_VAR to switch which config is loaded by the app is probably my preferred approach as long as I trust the people involved.


i.e. my concerns with config in the codebase are more about human issues than system / software design.


@maleghast as are all the issues I ever see!


Developer discipline conquers all!


Every poor codebase, badly maintained, bug ridden and/or inappropriate (to biz requirements) system I've ever worked with has got that way through misscomunication


I do like the idea of only baking in the config that's for the destination ENV when building Containers, so all the configs are in the codebase, only the correct one is deployed.


That's why the most important things we can do in software is recruit well, train and educate and talk!


@agile_geek - Yeah, it's certainly been issue #1 or #2 along with higher-up management dictat about time constraint


(leading to knowingly incurred technical debt that is then pathologically ignored / dismissed despite constant reminders)


@glenjamin wouldn't it make sense to do a ALLOW_HTTP instead of a HTTPS_ONLY. so that the less save option is something you have to make an exception for?


would make sense to me IMHO


only a matter of time before someone forgets to add that flag in production


@agile_geek > and then have an environment variable to tell the code which file to load This line I slightly disagree with. One file, no duplication is better. If you take a look at Aero, it gives you tools to do switching for particular variables & such, meaning there's no duplication (maintenance headache!)


@dominicm I can see the advantage of that


I like Aero too, use it a lot in Onyx jobs for :dev, :production and :mesos deploys


I work at juxt, we get whipped if we don't use it 😢 troll


I do think config around framework or structure of that app (which doesn't change per env) should be treated as code.


I guess my main objection is when ppl externalise things that change the way an app behaves. The naive assumption being that you can change the apps behaviour for new requirements without writing code (so externalised rules engines are the worst example). The issue being that you lose the disciplines required in development, i.e. config management, testing, etc.


@dominicm so.... when you start a new project at Juxt... do you start with a project.clj template that has every single Juxt lib in it?


@thomas possibly, it was a toss-up between ease of onboarding vs potential to forget in prod


Not to say you can't use a rules engine in an app...just make it a part of the app not a tool for business people to change code.


once it’s turned on once, there’s no reason to ever turn it off, so i figured prod wasn’t that risky


@glenjamin I think I know which of those two I would choose


but what if a new machine gets deployed, by someone who never has done it before?


that’s not really a thing these days is it?


new machine is incrementing the “number of nodes” counter by 1


it shouldn't... but I am sure it happens somewhere...


btw... not saying you made the wrong choice... just wondering what the trade off is.


@glenjamin hee hee...really? That's like saying the millions of lines of COBOL still in prod are not a thing these days 😈


well, this is a 12 factor app that isn’t deployed anywhere yet


what i mean is that a new build in my context won’t hit a manual-build-in-prod scenario


so that’s not a risk i’m concerned about


@agile_geek or what about the Cobol that still gets written today...


but I am now tempted to flip it over anyway


maybe you are lucky enough to work in an environment where you can do all those things .


i’d probably have the app load a .env file on startup which is git ignored in that case so i can easily set some dev stuff


maybe i’ll add a dummy config var called HAS_CONFIG which gives a nice friendly error message about how to set up config if its absent


no-one ever reads the setup docs, people almost never read warnings, but if the app refuses to start with an error message, it might sometimes get read

Rachel Westmacott09:02:17

^ this is a great approach (IMHO)


Doesn’t that approach make CI difficult difficult lemon difficult?


if you’re not checking in a config/env file then what does the likes of CircleCI do for testing?


suppose it depends on the nature of the config.


config is all from environment vars, the .env file is just a list of environment vars that get loaded


so in dev i don’t have to set them all the time


so, related question: the app talks to a backend API server, so one of the params is the URL to that Should it default to 🅰️ the non-prod API; 🅱️ the prod API or 🅾️ no default


@glenjamin There is a part of me that wants to say 🅰️ too


Particularly if it's a read-only API


Perhaps only if it's a read-only API


i generally have all my defaults for dev and require some effort to configure a production system


this is win-win, because everything in production is clustered and often requires a list of ips or names, so is going to require some config anyway

Rachel Westmacott10:02:15

I had voted "no default", but if the prod API is read-only and public/a conceptually separate service then (maybe only for some types of data?) it might make sense to default to the production API

Rachel Westmacott10:02:22

ie. if the backend API is essentially like a third party service


So, one of the nice things about "externalising" configuration is that people don't need to "know" clojure/C/whatever to use the tool. e.g, NGINX.

Rachel Westmacott10:02:08

that depends - e.g. you need a good knowledge of Java to write effective Spring XML configuration

Rachel Westmacott10:02:49

also, you then require people to know XML/JSON/YAML/whatever-they-think-of-next

Rachel Westmacott10:02:57

(admittedly less of an ask)

Rachel Westmacott10:02:09

(unless there's XSLT in your config pipeline)

Rachel Westmacott10:02:27

(in which case all is lost anyway, and you deserve whatever Cthulu sends)


When I have [cider/cider-nrepl “0.14.0”] and [refactor-nrepl “2.2.0”] in my plugins at the REPL (things I use to enable clojure-refactor moves) the REPL startup goes from 10secs to over 60secs. Anyone else experiencing the same?


I believe this is considered normal.


@glenjamin I would vote no default I think in that case.


yeah takes forever it feels @reborg


price to pay for the great tool then 😉


no default is friction for onboarding though 😞


> welcome to company X, download this config file from the wiki


trade of... just like anything else


In theory, it needn't be a trade-off. If those recent profilings prove to be fixable.

Rachel Westmacott10:02:46

@glenjamin can you provide no default in code but an easy way to include them in dev? e.g. My user ns reads an environment map from a file in the git repo to pass to the system on (reset) - but neither user namespace or defaults file is built into the app

Rachel Westmacott10:02:08

@reborg the longer the start-up time the more motivated one is to keep the same REPL session going - which is good for productivity in the long run...?


@glenjamin I'd prefer no default so long as the service doesn't fail to start, does produce obvious logs telling you what is missing and allows config reloading. And never prod as the default.


config reloading isn’t especially viable for this app, but it does start up in a few seconds so not a big deal


fwiw, my current approach is 🅰️

Rachel Westmacott10:02:30

I think as long as you've considered it and are aware of the trade-offs then its not worth losing sleep over


non-prod also fwiw, might consist of sit, stg, uat, dev etc all with perhaps different urls.


yeah, i’m not especially worried about any of the choices, just interested to see what people would pick


my usual approach is to default to things which make it easy to get a dev environment up and running


what is "non-prod" in this case @glenjamin ?


basically something that’s “safe” to use for dev


i think in this company they’ve called it beta


@dominicm good read thanks. So finally at the bottom somebody connected a profiler and find the offending ns. @peterwestmacott I’m with you. I’ve started around 10 REPLs 20 days ago and never shut them down. But I still start and stop many per days to try out things.

Rachel Westmacott11:02:57

@reborg - it was half in jest - I'm still excited when I learn new ways to un-bork a REPL


In my experience apps not starting due to a new feature being introduced and not having a default set wreaks havoc in promotion pipelines (i.e, don't have a default but also don't fail - dont' use the new feature at all). For instance if the app code is promoted to prod without the config change (as to whether it should be decoupled is another question - but they often are)

Rachel Westmacott11:02:11

Cursive has recently been slowing down a lot for me, I think in relation to REPLs or multiple windows, so I've had to kill a few recently. I've not invested the time to narrow down the cause.


@reborg this is actually why boot recommends not making cider activated by default, but explicitly starting with boot cider repl, so you can say "I'm rebooting a lot, and not using cider features.. zooom"


@dominicm good one thanks. Not on boot for many projects tho, but could use lein profiles and an alias in a similar way I suppose.


@reborg :knife: or force a move to boot


@mattford Yeah, i’ve seen that - my approach there has been to get devs involved in sending PRs to conf management code to get that config in there ahead of their code releases


yeah that sometimes works 🙂 but I've seen the dual. Unrecognised config options causing the app to bail => sad times.


yeah, that’s bad


same thing with extra properties in incoming API requests


Consul template for env specific app configuration and the app promotion job to update the key value store one possible solution


Back to Clojure; i have some functions that all require a profile variable. Rather than set it in every function I'd like to set it globally somehow. So I can write something like this:

(with-profile matt
   (dostuff add-perms)
   (dootherstuff 20 02))


Rather than

(dostuff matt add-perms)
etc etc


first answer: resist that urge, and pass it around


second answer: can you change the functions so they don’t need it?


in that particular example, -> might be useful


3rd answer: dynamic vars and binding (this one should be avoided though, prefer 1)


It's for an internal dsl. Seems to me having the context set (in this case it's an AWS profile) and having all the functions use it magically makes it read much more nicely. But I don't really know what I'm doing 🙂


try using the threading macros for that perhaps?


I shall go read up on all the answers 🙂 Cheers.


What's -> called?


"thread first” sometimes


thread-first macro


->> is thread-last


not sure it helps I don't have a pipeline as such


What I want is

(with-profile matt
   (dostuff add-perms)
   (dootherstuff 20 02))
to be similar to this
(let [dostuff* (partial dostuff matt)
      [dootherstuff* (partial dootherstuff matt)]
  (dostuff* add-perms)
  (dootherstuff* 20 02)))


Hopefully that makes sense.


But automagically over all dostuff functions.


(-> matt
  (dostuff add-perms)
  (dootherstuff 20 02))


expands to (dootherstuff (dostuff matt add-perms) 20 02)


yeah the dostuffs are completely unrelated


evaulated for side-effects only.


like ssh on box and run a command


mattford partial might also give you the movement towards fewer points too


fewer points?

Rachel Westmacott11:02:34

@mattford if you only want side-effects maybe doto instead of threading?


and it doesn't really solve your with-foo problem so I probably should have stayed quiet. 😊


((partial dostuff matt) add-perms)


live to learn


Question to ask yourself: If a fn needs something to do it's work is it better to explicitly declare that it uses that thing or to make that implicit? IMHO I favour being explicit. I makes understanding context and testing much easier. Therefore, I'm with @glenjamin just except that you pass it as an argument. YMMV


mattford agile_geek ^^


But my internal DSL is super ugly and very verbose if I have to go round putting profiles arguments to every function call. It gets worse the more context I have.


Perhaps I have the wrong approach.


(with-awsprofile dancingfrog
  (with-env sit ;; instances with env tag set to sit
    (stream-dropwizard-service alsmp) ;; instances with service tag set to alsmp
    (stream-dropwizard-service alspi)
    (stream-dropwizard-service dmdis)))


That's pretty to me 🙂


(stream-dropwizard-service dancingfrog sit alsmp)
(stream-dropwizard-service dancingfrog sit alspi)
(stream-dropwizard-service dancingfrog sit dmdis)
not so.


I think doto effectively does that


Perhaps have one map as your context with whatever keys for different bits


looks very promising!


I'd second the 'have one map as your context' suggestion


+1 context-map - maybe use your ioc container of choice to build your context-map and pass views of that around


any one here ever used Optimus ?


nope. but i would trust almost anything magnars does (if you are worried about quality)


Hello all. @thomas, I think it was you that mentioned the ProtoREPL talk last week. I finally got round to watching the whole thing. That is some really nice debugging feedback.


@benedek we are using it... I just don't really understand some of the behaviour we are seeing... debugging it at the moment and it is quite painful.


whoops, sorry @thomas, i guess my comment about quality become a bit ironic then...


I don't think it is a problem with Optimus... just not sure how I can explain the behaviour I am seeing...


some magic somewhere...