Fork me on GitHub
#clojure-uk
<
2017-02-02
>
dominicm07:02:22

Aero can do environment variables, but generally thinks you shouldn't :D

dominicm08:02:22

@paulspencerwilliams it's less obvious what your program is doing. Have you ever been confused as to why something as happening only to discover that is because of some environment thing (including files in odd places)

maleghast08:02:08

@dominicm I do know what you mean, but it's particularly useful when using containers to have config injected as ENVs and as long as it's documented it's really not that big a risk or mystery

maleghast08:02:52

@dominicm I'm just uncomfortable with config in the codebase these days, based on several bad experiences... ๐Ÿ˜‰

dominicm08:02:08

@maleghast most of the time you just need a variable to tell you you're in prod. And you can derive everything from there.

maleghast08:02:28

@dominicm I'm all for that if it works for you / your team / your codebases etc. I am just not ok with it, currently. I am prepared to be persuaded given more spare head-cycles to consider it...

agile_geek08:02:43

This "Where do I draw the line between code and data (re: properties files/environment variables)" is one that's perplexed me for over 20 years. Even before I moved to Clojure I had discovered thru using ORM's and frameworks that the 'config' is just as much a part of the behaviour of the system as the 'code'. The distinction is even less apparent in Clojure. In my experience if you change anything except trivial config you probably need to retest the entire app/component/system anyway so I tend to limit the externalisation of 'config' to things I absolutely know will change regularly in production and things that change in the journey to production (i.e. environment specific stuff like URLs and username/passwords). Anything complex I tend to encode in different files per environment (EDN is a good choice for format in Clojure or property files for Java etc.) and then have an environment variable to tell the code which file to load (i.e. an environment environment variable!).

agile_geek08:02:21

Basically config that changes behaviour is code

agile_geek08:02:41

So treat it that way: test it, version control it, etc.

agile_geek08:02:53

This is one area I really can speak form authority having written dozens of systems that were naively config driven and maintained many more. I've paid the price of the assumption that config makes systems easier to adapt than code. It's the other way around as you end up with lots of boiler plate to deal with decisions based on config and you always guess wrong and the thing you didn't configure for changes.

agile_geek08:02:17

Having said all that phew! you would think I'm anti framework....but I'm not! I think frameworks that use config are fine but the config must coexist happily with the code (not be externalised) and there must be tools and support to allow testing. So I'm interested in Arachne's DSL for config for example.

Rachel Westmacott08:02:23

"I am prepared to be persuaded given more spare head-cycles to consider it..." @maleghast I love this phrase.

Rachel Westmacott08:02:53

I think in the past there was this idea that you could redeploy the same code with different config files - but now people are all like: "let's put all the code and config down the same test-and-deploy-pipeline every time"

Rachel Westmacott08:02:19

I feel like it shouldn't really matter if you have lots of environmental factors that your code depends on - because your environment is also going to be built from code, so its just more code really.... plus some people really like their "12-factor apps"

agile_geek08:02:05

@maleghast me too. Not sure my argument has convinced you. Sounds like we have experiences at opposite ends of the scale! On a scale of maintainability nightmares->reasonable to maintain, I have worked with a half dozen case tools that generate code (words fail me for how bad these are), then written and used dozens of 'flexible, config driven/rules engine based systems' shudders, lots of systems with environment specific externalised config where everything that could be changed is external, and finally config packaged with code and a few simple environment variables to pick the right config. Much prefer the later.

Rachel Westmacott08:02:52

(having spoken in favour of env vars, I will just add that (obviously?) I think that minimising the surface area between the env and the code is going to be a good idea)

maleghast08:02:51

@agile_geek - Need time to digest this; am actually supposed to be working... Get back to you in a bit? (Sorry ๐Ÿ˜ž And thanks for the long post, I am interested ๐Ÿ™‚ )

Rachel Westmacott08:02:54

I like to think that researching coding best practices (ie. chatting on slack) is a key part of my job, and part of what makes me valuable. ๐Ÿ˜„

glenjamin08:02:56

config is stuff that changes depending on where you deploy

glenjamin08:02:02

anything else is just your app

glenjamin08:02:57

and RE: having loads of env vars, I try and resolve that with useful defaults

agile_geek08:02:07

@glenjamin I think you just summarised my rant to 3 sentences! simple_smile

glenjamin08:02:38

i pretty much agree with everything on that 12 factor page

glenjamin08:02:13

it might not all be perfect advice, but you need a damn good reason to deviate from it IME

agile_geek08:02:01

I would extend to say anything that has a lot of stuff that changes depending on where you deploy but doesn't change frequently should be packaged in your deployment unit with a simple env variable pointing to it.

glenjamin08:02:05

as a concrete example, the app iโ€™m currently working on takes an env var called HTTPS_ONLY. If you enable it you get a 301 redirect when accessing over http, HSTS headers and the cookies are all flagged as secure. Itโ€™s off by default so dev on localhost is easier

glenjamin08:02:27

meh, that stuff lives in my configuration management tool generally

agile_geek09:02:52

I guess that depends on the config and the CM tool but I have no problem with that.

maleghast09:02:57

@agile_geek - Right, read it, agree wholeheartedly, but my current situation is in a PHP environment and having the config in an entirely separate Git Repo, by environment (Dev, Stage, Prod) that has (until recent staff cuts) had much narrower access was the only way to stay sane. It is fair to say that the approach you describe, config per environment and then one ENV_VAR to switch which config is loaded by the app is probably my preferred approach as long as I trust the people involved.

maleghast09:02:19

i.e. my concerns with config in the codebase are more about human issues than system / software design.

agile_geek09:02:08

@maleghast as are all the issues I ever see!

agile_geek09:02:33

Developer discipline conquers all!

agile_geek09:02:07

Every poor codebase, badly maintained, bug ridden and/or inappropriate (to biz requirements) system I've ever worked with has got that way through misscomunication

maleghast09:02:41

I do like the idea of only baking in the config that's for the destination ENV when building Containers, so all the configs are in the codebase, only the correct one is deployed.

agile_geek09:02:42

That's why the most important things we can do in software is recruit well, train and educate and talk!

maleghast09:02:47

@agile_geek - Yeah, it's certainly been issue #1 or #2 along with higher-up management dictat about time constraint

maleghast09:02:27

(leading to knowingly incurred technical debt that is then pathologically ignored / dismissed despite constant reminders)

thomas09:02:30

@glenjamin wouldn't it make sense to do a ALLOW_HTTP instead of a HTTPS_ONLY. so that the less save option is something you have to make an exception for?

thomas09:02:19

would make sense to me IMHO

thomas09:02:40

only a matter of time before someone forgets to add that flag in production

dominicm09:02:42

@agile_geek > and then have an environment variable to tell the code which file to load This line I slightly disagree with. One file, no duplication is better. If you take a look at Aero, it gives you tools to do switching for particular variables & such, meaning there's no duplication (maintenance headache!)

agile_geek09:02:21

@dominicm I can see the advantage of that

jasonbell09:02:07

I like Aero too, use it a lot in Onyx jobs for :dev, :production and :mesos deploys

dominicm09:02:36

I work at juxt, we get whipped if we don't use it ๐Ÿ˜ข troll

agile_geek09:02:11

I do think config around framework or structure of that app (which doesn't change per env) should be treated as code.

agile_geek09:02:53

I guess my main objection is when ppl externalise things that change the way an app behaves. The naive assumption being that you can change the apps behaviour for new requirements without writing code (so externalised rules engines are the worst example). The issue being that you lose the disciplines required in development, i.e. config management, testing, etc.

thomas09:02:11

@dominicm so.... when you start a new project at Juxt... do you start with a project.clj template that has every single Juxt lib in it?

glenjamin09:02:46

@thomas possibly, it was a toss-up between ease of onboarding vs potential to forget in prod

agile_geek09:02:53

Not to say you can't use a rules engine in an app...just make it a part of the app not a tool for business people to change code.

glenjamin09:02:11

once itโ€™s turned on once, thereโ€™s no reason to ever turn it off, so i figured prod wasnโ€™t that risky

thomas09:02:33

@glenjamin I think I know which of those two I would choose

thomas09:02:28

but what if a new machine gets deployed, by someone who never has done it before?

glenjamin09:02:42

thatโ€™s not really a thing these days is it?

glenjamin09:02:57

new machine is incrementing the โ€œnumber of nodesโ€ counter by 1

thomas09:02:05

it shouldn't... but I am sure it happens somewhere...

thomas09:02:48

btw... not saying you made the wrong choice... just wondering what the trade off is.

agile_geek09:02:49

@glenjamin hee hee...really? That's like saying the millions of lines of COBOL still in prod are not a thing these days ๐Ÿ˜ˆ

glenjamin09:02:06

well, this is a 12 factor app that isnโ€™t deployed anywhere yet

glenjamin09:02:33

what i mean is that a new build in my context wonโ€™t hit a manual-build-in-prod scenario

glenjamin09:02:40

so thatโ€™s not a risk iโ€™m concerned about

thomas09:02:42

@agile_geek or what about the Cobol that still gets written today...

glenjamin09:02:09

but I am now tempted to flip it over anyway

thomas09:02:18

maybe you are lucky enough to work in an environment where you can do all those things .

glenjamin09:02:40

iโ€™d probably have the app load a .env file on startup which is git ignored in that case so i can easily set some dev stuff

glenjamin09:02:25

maybe iโ€™ll add a dummy config var called HAS_CONFIG which gives a nice friendly error message about how to set up config if its absent

glenjamin09:02:13

no-one ever reads the setup docs, people almost never read warnings, but if the app refuses to start with an error message, it might sometimes get read

Rachel Westmacott09:02:17

^ this is a great approach (IMHO)

jasonbell09:02:45

Doesnโ€™t that approach make CI difficult difficult lemon difficult?

jasonbell09:02:54

if youโ€™re not checking in a config/env file then what does the likes of CircleCI do for testing?

jasonbell09:02:14

suppose it depends on the nature of the config.

glenjamin09:02:19

config is all from environment vars, the .env file is just a list of environment vars that get loaded

glenjamin09:02:37

so in dev i donโ€™t have to set them all the time

glenjamin10:02:11

so, related question: the app talks to a backend API server, so one of the params is the URL to that Should it default to ๐Ÿ…ฐ๏ธ the non-prod API; ๐Ÿ…ฑ๏ธ the prod API or ๐Ÿ…พ๏ธ no default

maleghast10:02:50

@glenjamin There is a part of me that wants to say ๐Ÿ…ฐ๏ธ too

maleghast10:02:07

Particularly if it's a read-only API

maleghast10:02:25

Perhaps only if it's a read-only API

mccraigmccraig10:02:32

i generally have all my defaults for dev and require some effort to configure a production system

mccraigmccraig10:02:51

this is win-win, because everything in production is clustered and often requires a list of ips or names, so is going to require some config anyway

Rachel Westmacott10:02:15

I had voted "no default", but if the prod API is read-only and public/a conceptually separate service then (maybe only for some types of data?) it might make sense to default to the production API

Rachel Westmacott10:02:22

ie. if the backend API is essentially like a third party service

mattford10:02:24

So, one of the nice things about "externalising" configuration is that people don't need to "know" clojure/C/whatever to use the tool. e.g, NGINX.

Rachel Westmacott10:02:08

that depends - e.g. you need a good knowledge of Java to write effective Spring XML configuration

Rachel Westmacott10:02:49

also, you then require people to know XML/JSON/YAML/whatever-they-think-of-next

Rachel Westmacott10:02:57

(admittedly less of an ask)

Rachel Westmacott10:02:09

(unless there's XSLT in your config pipeline)

Rachel Westmacott10:02:27

(in which case all is lost anyway, and you deserve whatever Cthulu sends)

reborg10:02:13

When I have [cider/cider-nrepl โ€œ0.14.0โ€] and [refactor-nrepl โ€œ2.2.0โ€] in my plugins at the REPL (things I use to enable clojure-refactor moves) the REPL startup goes from 10secs to over 60secs. Anyone else experiencing the same?

dominicm10:02:37

I believe this is considered normal.

thomas10:02:01

@glenjamin I would vote no default I think in that case.

mattford10:02:10

yeah takes forever it feels @reborg

reborg10:02:32

price to pay for the great tool then ๐Ÿ˜‰

glenjamin10:02:33

no default is friction for onboarding though ๐Ÿ˜ž

glenjamin10:02:45

> welcome to company X, download this config file from the wiki

thomas10:02:01

trade of... just like anything else

dominicm10:02:35

In theory, it needn't be a trade-off. If those recent profilings prove to be fixable.

Rachel Westmacott10:02:46

@glenjamin can you provide no default in code but an easy way to include them in dev? e.g. My user ns reads an environment map from a file in the git repo to pass to the system on (reset) - but neither user namespace or defaults file is built into the app

Rachel Westmacott10:02:08

@reborg the longer the start-up time the more motivated one is to keep the same REPL session going - which is good for productivity in the long run...?

mattford10:02:09

@glenjamin I'd prefer no default so long as the service doesn't fail to start, does produce obvious logs telling you what is missing and allows config reloading. And never prod as the default.

glenjamin10:02:43

config reloading isnโ€™t especially viable for this app, but it does start up in a few seconds so not a big deal

glenjamin10:02:54

fwiw, my current approach is ๐Ÿ…ฐ๏ธ

Rachel Westmacott10:02:30

I think as long as you've considered it and are aware of the trade-offs then its not worth losing sleep over

mattford10:02:47

non-prod also fwiw, might consist of sit, stg, uat, dev etc all with perhaps different urls.

glenjamin10:02:53

yeah, iโ€™m not especially worried about any of the choices, just interested to see what people would pick

glenjamin10:02:25

my usual approach is to default to things which make it easy to get a dev environment up and running

mccraigmccraig10:02:32

what is "non-prod" in this case @glenjamin ?

glenjamin11:02:20

basically something thatโ€™s โ€œsafeโ€ to use for dev

glenjamin11:02:30

i think in this company theyโ€™ve called it beta

reborg11:02:58

@dominicm good read thanks. So finally at the bottom somebody connected a profiler and find the offending ns. @peterwestmacott Iโ€™m with you. Iโ€™ve started around 10 REPLs 20 days ago and never shut them down. But I still start and stop many per days to try out things.

Rachel Westmacott11:02:57

@reborg - it was half in jest - I'm still excited when I learn new ways to un-bork a REPL

mattford11:02:59

In my experience apps not starting due to a new feature being introduced and not having a default set wreaks havoc in promotion pipelines (i.e, don't have a default but also don't fail - dont' use the new feature at all). For instance if the app code is promoted to prod without the config change (as to whether it should be decoupled is another question - but they often are)

Rachel Westmacott11:02:11

Cursive has recently been slowing down a lot for me, I think in relation to REPLs or multiple windows, so I've had to kill a few recently. I've not invested the time to narrow down the cause.

dominicm11:02:34

@reborg this is actually why boot recommends not making cider activated by default, but explicitly starting with boot cider repl, so you can say "I'm rebooting a lot, and not using cider features.. zooom"

reborg11:02:47

@dominicm good one thanks. Not on boot for many projects tho, but could use lein profiles and an alias in a similar way I suppose.

dominicm11:02:23

@reborg :knife: or force a move to boot

glenjamin11:02:14

@mattford Yeah, iโ€™ve seen that - my approach there has been to get devs involved in sending PRs to conf management code to get that config in there ahead of their code releases

mattford11:02:01

yeah that sometimes works ๐Ÿ™‚ but I've seen the dual. Unrecognised config options causing the app to bail => sad times.

glenjamin11:02:49

yeah, thatโ€™s bad

glenjamin11:02:02

same thing with extra properties in incoming API requests

mattford11:02:16

Consul template for env specific app configuration and the app promotion job to update the key value store one possible solution https://github.com/hashicorp/consul-template

mattford11:02:23

Back to Clojure; i have some functions that all require a profile variable. Rather than set it in every function I'd like to set it globally somehow. So I can write something like this:

(with-profile matt
   (dostuff add-perms)
   (dootherstuff 20 02))

mattford11:02:06

Rather than

(dostuff matt add-perms)
etc etc

glenjamin11:02:09

first answer: resist that urge, and pass it around

glenjamin11:02:23

second answer: can you change the functions so they donโ€™t need it?

glenjamin11:02:44

in that particular example, -> might be useful

dominicm11:02:24

3rd answer: dynamic vars and binding (this one should be avoided though, prefer 1)

mattford11:02:50

It's for an internal dsl. Seems to me having the context set (in this case it's an AWS profile) and having all the functions use it magically makes it read much more nicely. But I don't really know what I'm doing ๐Ÿ™‚

glenjamin11:02:12

try using the threading macros for that perhaps?

mattford11:02:46

I shall go read up on all the answers ๐Ÿ™‚ Cheers.

mattford11:02:57

What's -> called?

glenjamin11:02:39

"thread firstโ€ sometimes

maleghast11:02:12

thread-first macro

maleghast11:02:51

->> is thread-last

mattford11:02:20

not sure it helps I don't have a pipeline as such

mattford11:02:45

What I want is

(with-profile matt
   (dostuff add-perms)
   (dootherstuff 20 02))
to be similar to this
(let [dostuff* (partial dostuff matt)
      [dootherstuff* (partial dootherstuff matt)]
  (dostuff* add-perms)
  (dootherstuff* 20 02)))

mattford11:02:13

Hopefully that makes sense.

mattford11:02:50

But automagically over all dostuff functions.

glenjamin11:02:20

(-> matt
  (dostuff add-perms)
  (dootherstuff 20 02))

glenjamin11:02:41

expands to (dootherstuff (dostuff matt add-perms) 20 02)

mattford11:02:12

yeah the dostuffs are completely unrelated

mattford11:02:20

evaulated for side-effects only.

mattford11:02:31

like ssh on box and run a command

otfrom11:02:39

mattford partial might also give you the movement towards fewer points too

mattford11:02:17

fewer points?

Rachel Westmacott11:02:34

@mattford if you only want side-effects maybe doto instead of threading?

otfrom11:02:49

and it doesn't really solve your with-foo problem so I probably should have stayed quiet. ๐Ÿ˜Š

otfrom11:02:11

((partial dostuff matt) add-perms)

mattford12:02:10

live to learn

agile_geek12:02:28

Question to ask yourself: If a fn needs something to do it's work is it better to explicitly declare that it uses that thing or to make that implicit? IMHO I favour being explicit. I makes understanding context and testing much easier. Therefore, I'm with @glenjamin just except that you pass it as an argument. YMMV

otfrom12:02:13

mattford agile_geek ^^

mattford12:02:35

But my internal DSL is super ugly and very verbose if I have to go round putting profiles arguments to every function call. It gets worse the more context I have.

mattford12:02:38

Perhaps I have the wrong approach.

mattford12:02:07

(with-awsprofile dancingfrog
  (with-env sit ;; instances with env tag set to sit
    (stream-dropwizard-service alsmp) ;; instances with service tag set to alsmp
    (stream-dropwizard-service alspi)
    (stream-dropwizard-service dmdis)))

mattford12:02:51

That's pretty to me ๐Ÿ™‚

mattford12:02:56

(stream-dropwizard-service dancingfrog sit alsmp)
(stream-dropwizard-service dancingfrog sit alspi)
(stream-dropwizard-service dancingfrog sit dmdis)
not so.

glenjamin12:02:53

I think doto effectively does that

glenjamin12:02:26

Perhaps have one map as your context with whatever keys for different bits

mattford12:02:51

looks very promising!

agile_geek14:02:33

I'd second the 'have one map as your context' suggestion

mccraigmccraig14:02:50

+1 context-map - maybe use your ioc container of choice to build your context-map and pass views of that around

thomas15:02:24

any one here ever used Optimus ?

benedek16:02:14

nope. but i would trust almost anything magnars does (if you are worried about quality)

yogidevbear16:02:56

Hello all. @thomas, I think it was you that mentioned the ProtoREPL talk last week. I finally got round to watching the whole thing. That is some really nice debugging feedback.

thomas16:02:30

@benedek we are using it... I just don't really understand some of the behaviour we are seeing... debugging it at the moment and it is quite painful.

benedek16:02:34

whoops, sorry @thomas, i guess my comment about quality become a bit ironic then...

thomas16:02:31

I don't think it is a problem with Optimus... just not sure how I can explain the behaviour I am seeing...

thomas16:02:41

some magic somewhere...