This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-02
Channels
- # aws-lambda (1)
- # beginners (46)
- # boot (190)
- # cider (12)
- # clara (6)
- # cljs-dev (9)
- # cljsjs (8)
- # clojure (152)
- # clojure-austin (3)
- # clojure-berlin (3)
- # clojure-finland (2)
- # clojure-france (5)
- # clojure-italy (3)
- # clojure-russia (92)
- # clojure-serbia (4)
- # clojure-spec (7)
- # clojure-uk (190)
- # clojurescript (115)
- # cursive (20)
- # datomic (20)
- # dirac (4)
- # emacs (9)
- # gsoc (5)
- # hoplon (1)
- # jobs (1)
- # klipse (4)
- # lein-figwheel (1)
- # leiningen (6)
- # lumo (2)
- # mount (18)
- # off-topic (57)
- # om (68)
- # om-next (14)
- # onyx (33)
- # perun (32)
- # portland-or (4)
- # re-frame (21)
- # reagent (85)
- # ring (6)
- # ring-swagger (23)
- # schema (1)
- # uncomplicate (1)
- # untangled (13)
- # vim (7)
Morning.
@dominicm: why not?
@paulspencerwilliams it's less obvious what your program is doing. Have you ever been confused as to why something as happening only to discover that is because of some environment thing (including files in odd places)
Bore da
@dominicm I do know what you mean, but it's particularly useful when using containers to have config injected as ENVs and as long as it's documented it's really not that big a risk or mystery
@dominicm I'm just uncomfortable with config in the codebase these days, based on several bad experiences... ๐
@maleghast most of the time you just need a variable to tell you you're in prod. And you can derive everything from there.
@dominicm I'm all for that if it works for you / your team / your codebases etc. I am just not ok with it, currently. I am prepared to be persuaded given more spare head-cycles to consider it...
This "Where do I draw the line between code and data (re: properties files/environment variables)" is one that's perplexed me for over 20 years. Even before I moved to Clojure I had discovered thru using ORM's and frameworks that the 'config' is just as much a part of the behaviour of the system as the 'code'. The distinction is even less apparent in Clojure. In my experience if you change anything except trivial config you probably need to retest the entire app/component/system anyway so I tend to limit the externalisation of 'config' to things I absolutely know will change regularly in production and things that change in the journey to production (i.e. environment specific stuff like URLs and username/passwords). Anything complex I tend to encode in different files per environment (EDN is a good choice for format in Clojure or property files for Java etc.) and then have an environment variable to tell the code which file to load (i.e. an environment environment variable!).
Basically config that changes behaviour is code
So treat it that way: test it, version control it, etc.
This is one area I really can speak form authority having written dozens of systems that were naively config driven and maintained many more. I've paid the price of the assumption that config makes systems easier to adapt than code. It's the other way around as you end up with lots of boiler plate to deal with decisions based on config and you always guess wrong and the thing you didn't configure for changes.
Having said all that phew! you would think I'm anti framework....but I'm not! I think frameworks that use config are fine but the config must coexist happily with the code (not be externalised) and there must be tools and support to allow testing. So I'm interested in Arachne's DSL for config for example.
"I am prepared to be persuaded given more spare head-cycles to consider it..." @maleghast I love this phrase.
I think in the past there was this idea that you could redeploy the same code with different config files - but now people are all like: "let's put all the code and config down the same test-and-deploy-pipeline every time"
I feel like it shouldn't really matter if you have lots of environmental factors that your code depends on - because your environment is also going to be built from code, so its just more code really.... plus some people really like their "12-factor apps"
@maleghast me too. Not sure my argument has convinced you. Sounds like we have experiences at opposite ends of the scale! On a scale of maintainability nightmares->reasonable to maintain, I have worked with a half dozen case tools that generate code (words fail me for how bad these are), then written and used dozens of 'flexible, config driven/rules engine based systems' shudders, lots of systems with environment specific externalised config where everything that could be changed is external, and finally config packaged with code and a few simple environment variables to pick the right config. Much prefer the later.
(having spoken in favour of env vars, I will just add that (obviously?) I think that minimising the surface area between the env and the code is going to be a good idea)
@agile_geek - Need time to digest this; am actually supposed to be working... Get back to you in a bit? (Sorry ๐ And thanks for the long post, I am interested ๐ )
I like to think that researching coding best practices (ie. chatting on slack) is a key part of my job, and part of what makes me valuable. ๐
@glenjamin I think you just summarised my rant to 3 sentences!
it might not all be perfect advice, but you need a damn good reason to deviate from it IME
I would extend to say anything that has a lot of stuff that changes depending on where you deploy but doesn't change frequently should be packaged in your deployment unit with a simple env variable pointing to it.
as a concrete example, the app iโm currently working on takes an env var called HTTPS_ONLY
. If you enable it you get a 301 redirect when accessing over http, HSTS headers and the cookies are all flagged as secure. Itโs off by default so dev on localhost is easier
I guess that depends on the config and the CM tool but I have no problem with that.
@agile_geek - Right, read it, agree wholeheartedly, but my current situation is in a PHP environment and having the config in an entirely separate Git Repo, by environment (Dev, Stage, Prod) that has (until recent staff cuts) had much narrower access was the only way to stay sane. It is fair to say that the approach you describe, config per environment and then one ENV_VAR to switch which config is loaded by the app is probably my preferred approach as long as I trust the people involved.
i.e. my concerns with config in the codebase are more about human issues than system / software design.
@maleghast as are all the issues I ever see!
Developer discipline conquers all!
Every poor codebase, badly maintained, bug ridden and/or inappropriate (to biz requirements) system I've ever worked with has got that way through misscomunication
I do like the idea of only baking in the config that's for the destination ENV when building Containers, so all the configs are in the codebase, only the correct one is deployed.
That's why the most important things we can do in software is recruit well, train and educate and talk!
@agile_geek - Yeah, it's certainly been issue #1 or #2 along with higher-up management dictat about time constraint
(leading to knowingly incurred technical debt that is then pathologically ignored / dismissed despite constant reminders)
@glenjamin wouldn't it make sense to do a ALLOW_HTTP
instead of a HTTPS_ONLY
. so that the less save option is something you have to make an exception for?
@thomas I was wondering that
(morning, btw)
@agile_geek > and then have an environment variable to tell the code which file to load This line I slightly disagree with. One file, no duplication is better. If you take a look at Aero, it gives you tools to do switching for particular variables & such, meaning there's no duplication (maintenance headache!)
@dominicm I can see the advantage of that
I like Aero too, use it a lot in Onyx jobs for :dev
, :production
and :mesos
deploys
I do think config around framework or structure of that app (which doesn't change per env) should be treated as code.
I guess my main objection is when ppl externalise things that change the way an app behaves. The naive assumption being that you can change the apps behaviour for new requirements without writing code (so externalised rules engines are the worst example). The issue being that you lose the disciplines required in development, i.e. config management, testing, etc.
@dominicm so.... when you start a new project at Juxt... do you start with a project.clj
template that has every single Juxt lib in it?
@thomas possibly, it was a toss-up between ease of onboarding vs potential to forget in prod
Not to say you can't use a rules engine in an app...just make it a part of the app not a tool for business people to change code.
once itโs turned on once, thereโs no reason to ever turn it off, so i figured prod wasnโt that risky
@glenjamin I think I know which of those two I would choose
@glenjamin hee hee...really? That's like saying the millions of lines of COBOL still in prod are not a thing these days ๐
what i mean is that a new build in my context wonโt hit a manual-build-in-prod scenario
@agile_geek or what about the Cobol that still gets written today...
fair enough @glenjamin
maybe you are lucky enough to work in an environment where you can do all those things .
iโd probably have the app load a .env
file on startup which is git ignored in that case so i can easily set some dev stuff
maybe iโll add a dummy config var called HAS_CONFIG
which gives a nice friendly error message about how to set up config if its absent
no-one ever reads the setup docs, people almost never read warnings, but if the app refuses to start with an error message, it might sometimes get read
^ this is a great approach (IMHO)
if youโre not checking in a config/env file then what does the likes of CircleCI do for testing?
config is all from environment vars, the .env
file is just a list of environment vars that get loaded
so, related question: the app talks to a backend API server, so one of the params is the URL to that Should it default to ๐ ฐ๏ธ the non-prod API; ๐ ฑ๏ธ the prod API or ๐ พ๏ธ no default
@glenjamin There is a part of me that wants to say ๐ ฐ๏ธ too
i generally have all my defaults for dev and require some effort to configure a production system
this is win-win, because everything in production is clustered and often requires a list of ips or names, so is going to require some config anyway
I had voted "no default", but if the prod API is read-only and public/a conceptually separate service then (maybe only for some types of data?) it might make sense to default to the production API
ie. if the backend API is essentially like a third party service
So, one of the nice things about "externalising" configuration is that people don't need to "know" clojure/C/whatever to use the tool. e.g, NGINX.
that depends - e.g. you need a good knowledge of Java to write effective Spring XML configuration
also, you then require people to know XML/JSON/YAML/whatever-they-think-of-next
(admittedly less of an ask)
(unless there's XSLT in your config pipeline)
(in which case all is lost anyway, and you deserve whatever Cthulu sends)
When I have [cider/cider-nrepl โ0.14.0โ]
and [refactor-nrepl โ2.2.0โ]
in my plugins at the REPL (things I use to enable clojure-refactor moves) the REPL startup goes from 10secs to over 60secs. Anyone else experiencing the same?
@glenjamin I would vote no default I think in that case.
In theory, it needn't be a trade-off. If those recent profilings prove to be fixable.
@glenjamin can you provide no default in code but an easy way to include them in dev? e.g. My user
ns reads an environment map from a file in the git repo to pass to the system on (reset)
- but neither user namespace or defaults file is built into the app
@reborg the longer the start-up time the more motivated one is to keep the same REPL session going - which is good for productivity in the long run...?
@glenjamin I'd prefer no default so long as the service doesn't fail to start, does produce obvious logs telling you what is missing and allows config reloading. And never prod as the default.
config reloading isnโt especially viable for this app, but it does start up in a few seconds so not a big deal
I think as long as you've considered it and are aware of the trade-offs then its not worth losing sleep over
non-prod also fwiw, might consist of sit, stg, uat, dev etc all with perhaps different urls.
yeah, iโm not especially worried about any of the choices, just interested to see what people would pick
my usual approach is to default to things which make it easy to get a dev environment up and running
what is "non-prod" in this case @glenjamin ?
@dominicm good read thanks. So finally at the bottom somebody connected a profiler and find the offending ns. @peterwestmacott Iโm with you. Iโve started around 10 REPLs 20 days ago and never shut them down. But I still start and stop many per days to try out things.
@reborg - it was half in jest - I'm still excited when I learn new ways to un-bork a REPL
In my experience apps not starting due to a new feature being introduced and not having a default set wreaks havoc in promotion pipelines (i.e, don't have a default but also don't fail - dont' use the new feature at all). For instance if the app code is promoted to prod without the config change (as to whether it should be decoupled is another question - but they often are)
Cursive has recently been slowing down a lot for me, I think in relation to REPLs or multiple windows, so I've had to kill a few recently. I've not invested the time to narrow down the cause.
@reborg this is actually why boot recommends not making cider activated by default, but explicitly starting with boot cider repl
, so you can say "I'm rebooting a lot, and not using cider features.. zooom"
@dominicm good one thanks. Not on boot for many projects tho, but could use lein profiles and an alias in a similar way I suppose.
@mattford Yeah, iโve seen that - my approach there has been to get devs involved in sending PRs to conf management code to get that config in there ahead of their code releases
yeah that sometimes works ๐ but I've seen the dual. Unrecognised config options causing the app to bail => sad times.
Consul template for env specific app configuration and the app promotion job to update the key value store one possible solution https://github.com/hashicorp/consul-template
Back to Clojure; i have some functions that all require a profile variable. Rather than set it in every function I'd like to set it globally somehow. So I can write something like this:
(with-profile matt
(dostuff add-perms)
(dootherstuff 20 02))
It's for an internal dsl. Seems to me having the context set (in this case it's an AWS profile) and having all the functions use it magically makes it read much more nicely. But I don't really know what I'm doing ๐
What I want is
(with-profile matt
(dostuff add-perms)
(dootherstuff 20 02))
to be similar to this (let [dostuff* (partial dostuff matt)
[dootherstuff* (partial dootherstuff matt)]
(dostuff* add-perms)
(dootherstuff* 20 02)))
@mattford if you only want side-effects maybe doto
instead of threading?
and it doesn't really solve your with-foo problem so I probably should have stayed quiet. ๐
Interesting all the same @otfrom https://en.wikipedia.org/wiki/Tacit_programming
Question to ask yourself: If a fn needs something to do it's work is it better to explicitly declare that it uses that thing or to make that implicit? IMHO I favour being explicit. I makes understanding context and testing much easier. Therefore, I'm with @glenjamin just except that you pass it as an argument. YMMV
But my internal DSL is super ugly and very verbose if I have to go round putting profiles arguments to every function call. It gets worse the more context I have.
(with-awsprofile dancingfrog
(with-env sit ;; instances with env tag set to sit
(stream-dropwizard-service alsmp) ;; instances with service tag set to alsmp
(stream-dropwizard-service alspi)
(stream-dropwizard-service dmdis)))
(stream-dropwizard-service dancingfrog sit alsmp)
(stream-dropwizard-service dancingfrog sit alspi)
(stream-dropwizard-service dancingfrog sit dmdis)
not so.I'd second the 'have one map as your context' suggestion
+1 context-map - maybe use your ioc container of choice to build your context-map and pass views of that around
nope. but i would trust almost anything magnars does (if you are worried about quality)
Hello all. @thomas, I think it was you that mentioned the ProtoREPL talk last week. I finally got round to watching the whole thing. That is some really nice debugging feedback.
@benedek we are using it... I just don't really understand some of the behaviour we are seeing... debugging it at the moment and it is quite painful.