Clojurians Log v2

So the method I like is to: • Commit a well-formed plaintext version of config, with values used by local test runners. This makes it easy to revise + code review config changes. • Make the app startup sequence decrypt (for non-local environments) and validate the config file (use spec / schema / malli etc.) • Keep the encrypted prod secret file in a secure location accessible only to prod scripts • Keep a plaintext CI version also in central control, with different values from local source • Never package the secret file in anything This way I can a) structure explicit change management of configuration and secrets and b) add safety by making deployments to CI or staging fail (or worst case, fail prod deployment) if someone updates local config schema, but not the CI and/or secret versions.

Approximately like this: https://gitlab.com/nilenso/cats/-/blob/master/resources/config.edn?ref_type=heads, a https://gitlab.com/nilenso/cats/-/blob/master/src/clj/cats/config.clj?ref_type=heads#L33 configuration gate at startup, and a https://gitlab.com/nilenso/cats/-/blob/master/bin/cicd-deployment.sh?ref_type=heads#L90 (knows almost nothing about the app). (The https://gitlab.com/nilenso/cats/-/commit/478d2b86ed8caae74584cad279d1c570926f8627 explains the reasoning. Please excuse code quality... it is a stale study repo).

The issue with a database for configuration, is that it's out of sync with the code.

It's undeniable that things pan out the way they do historically for good reasons though.

Yes, but sometimes rolling back the code, but keeping the current configuration would be desirable right?

Having some statistics on usage patterns might help make an informed decision.

I feel it can be reasoned from first principle too in this case. You deployed, it broke. You rollback to last working state. If you "need" the new config, then that config change itself was for a bug fix. But it does not work with new code, your fix don't work. Maybe old code + new config would work, but you did not test that. You can also off course, cherry pick only config change, over new code, and push that too.

All of this flexibility in a DB, just doesn't exist

Bold statement.

Or, what some people do, is their config is in it's own Git repo, and pulled in as a library. Therefore it is versioned, and in the service it depends on it and specifies the version. So you can independently pick the config version you want. It's not much different, but it's a bit easier to mix and match a code commit with a config version. And still has all the benefits I mentioned.

Is it? What DB offers PR review before committing a change with a diff view? What DB versions all changes so you can rollback to last DB state? What DB handles merge conflicts?

Another issue with config in DB, is reproducible build. Since the DB is mutable and always just in the current state, it's hard to reproduce what was running in prod exactly at some point in time to root cause.

By the way, I'm not talking about 12-factor app style config. Environment Variables also have all the same issues. What you want is the config in code. You give your environments a name, and in code, have a config for each named environment. When you load the config, you load the one for the name of the environment, but the config is in the code

{dev: {}
 alpha: {}
 beta: {}
 staging: {}
 prod: {}}

Sorry, didn't want to be dismissive. There might be some reverse scenario as you describe, I don't have large scale data, just my own experiential data, which could be biased by coincidence. That said, I do think logically, it also seems to point to what I'm saying. From a pros/cons list, a DB lacks quite a lot of features to manage safe config changes and rollbacks.

What database offers a preview of the changes it's going to make?

That's what I mean, I don't know of any. So git seems much more appropriate.

Datomic... Yeah?

Well, AWS AppConfig actually might. Not sure if it counts as a DB, but it's specifically meant to manage config.

Never used datomic 😞

There is a function where it will return you a diff and show the conflicts?

Think of it from a change management point of view. I'm working locally on my feature. I need some new config, or to change some config. I would modify it into either a local Datomic instance, or the prod Datomic instance but under some partition that has my username. Now locally my code is using my config. I work on my feature. In parallel, other people are doing the same. Then I want to push my feature to beta. And I need to promote the config changes I made to my partition of Datomic to the beta Datomic? Is that even possible? Or I need to remember all changes and re-run them on Datomic? Or am I expected to like code up a script for a up/down change to the DB related to my config changes? And now if my config changes conflict with some someone else promoted to beta before me?

And how do I get someone else to review my config changes?

by sharing them? You can encode and write down transactions. They need not be issued in a terminal you discard.

Sharing them... like in code? In git?

yeah.

You can setup a git hook if you like around an acceptance process.

I also feel, it's much harder to detect conflict if I have: transacrt(update X to 10) transact(update X to 12) These won't conflict in git, since I assume they would be in different files/places. But if you had:

{:x 10}

{:x 12}

What do you mean conflict? There is no conflict on the transaction itself there.

It also seems to defeat the point of putting it in a DB? What's the benefit now? It behaves the same as if it was just stored in code no?

Conflict as in, two engineer want X to be configured differently.

So one of their feature/bugfix won't work :d

there is no general solution to that issue though, git doesn't solve that either.

a git conflict is when git doesn't know how to merge the stuff (idk text?)

not a way to merge the functionality.

Git detects the conflict. And fails the merge for the person that lost the race. Now they have to rework their approach so it works with the current mainline.

With what you suggest, it seems the issue would be caught after the merge, maybe when it goes to staging, or even prod

git doesn't' detect conflicts in resolving business level issues at all.

It's not business level, this is infra config

Say for example, I made the thread pool size 20, and you made it 8. We both started thinking it was currently set to 1. The last person to merge will get a conflict. And they will have to ask themselves... hum ok, why did this other person make it 8? Maybe I'll talk to them. Is it an issue if I make it 20, and so on.

Wait... Are you talking business level config? Like user preferences and so on?

they don't have to ask themselves that at all, the can force push it 😈

Sure, but they are making the decision to force push, knowing there was a conflict. The other way, they don't even know there is a conflict. So they can't ask themselves, should I force it? Should I inquire?

again, there is no conflict in changing, or example, x from 10 to 12. Git wouldn't even show a conflict. It would show the change, but not a conflict right?

Ya git shows conflicts when two people change the same line of code.

Because your changes applied to an earlier commit, you will have to pull. And the pull will do a 3-way merge.

When you pull, it will detect that both you and the commits being pulled are trying to change the same lines. And it will ask you to resolve the conflict.

Right, well, when you go to transact to the database you can check the current value as well right?

Let's go back a bit though. In the scenario where changes are applied to the config DB as part of your CI/CD pipeline automatically. Where you use up/down transaction scripts to update the config in the DB and roll it back. What do you gain with this setup as opposed to just having the config in the code?

What do you gain with this setup as opposed to just having the config in the code?

Like with most things, starting small and building up is a good idea. In this case, I would follow the historical precedent, but I think at some point in the narrative people start re-inventing poor solutions rather then adopting conventions in an orthogonal domain and successful navigating the middle ground is about identifying how to ease the transition. You identified a lot of common things people want when dealing with configuration. 🙂. In part, i think a lot of applications dont' become as configurable as they could be, because it's hard to manage the data that configures them. Like if it was common to keep business data in a flat file, as opposed to a relational database, they would be keen to see solutions that fit that tool.

I need to run. 🏃 I might be able to check this in a week when i get back -_-.

From my point of view, the historical precent for config, is storing it in code....

> Query and transaction semantics, ACID guarantees, the ability to have post and pre-hook middle ware on your database, schema checks, etc.. Querying EDN is pretty simple, not sure I need datalog just to lookup: (get-in config [env :db-url]) . You also would not do that, you would not want to query the DB every time you need the config value, you'd query the whole config once and cache it in your running instance. Config in code is also ACID compliant. post/pre hook middleware, you need to tell me what you want to do with that? I don't follow. Schema check I'd argue are better at compile time here. If your config is in code, you can have a unit test validate it at build. I normally use Spec. > If you have 9 applications sharing the same set of configuration, with changes here and there, across 3 environments and you want to update or query them then you end up wanting ... That sounds like a recipe for waking up at 3am... I would never share application config between apps. Have your own config per-app, copy/paste if you want to save some time initially. Don't risk a config change for one app take down 8 other services.

I suspect maybe we're not thinking of the same kind of configuration? I'm talking about configuring thread pools, client endpoints, DB endpoints, logging directory, table names, etc.

For me, a lot of this comes down to the same sort of considerations of whether you have SQL code in your app or all in stored procs in the DB. I've met people who insist on stored procs all the way but to me you lose so much leverage that our entire code/change management stack had provided.

At work, we have two types of "configuration": infra is all code (and edn files); app metadata is all database. And that really comes down to whose responsibility should it be to apply changes? For us, if the business folks might want to change a setting - or some descriptive data for the domain - then it goes in the DB and we build them an admin UI for it. Over the years we've gotten better at making systems robust enough to business-driven changes that we try to default to the database, with an admin UI, for most new settings, and some of the domain logic is driven by rules that are English-like so the business team can write their own rules. But they can't change any infrastructure configuration.

App metadata is a good name. I tend to think of it as user customization/configuration. If you assume the users are the business teams. Though it gets confusing with having end users of the app, and internal business users.

as for config items you can't just rolback in case of a broken deployment there are expired secrets, e.g. tokens or certificates. You want to keep the new ones even in the case of a rollback.

Those shouldn't be in config anyways no? At least I don't store those in config. They get deployed by docker, kubernetes, or some secret manager system in place.

Quite an interesting discussion in this thread, there are interesting tradeoffs between the different ways to store configuration. I've worked with old software that stored most of its configuration in a database and it had a gui to configure it. The reason for those feature was the convenience for the clients who had to reconfigure once in while; I know of few instances where that actually happened though and the subsequent rewrites on that platform stored config in json and eventually yaml files..

I am really happy with AWS secrets manager as a sort of DB for config

I would assume micro services would lead to a resurgence of configuration in a db, but it seems it let to resurgence of ENV vars instead lol

Oh that old software I worked with had a hexagonal (ports and adapters) architecture and in theory could to store a ton of configuration for various integrations (it was an EFT system). Pretty much the patter Rupert mentioned above

What if the answer is we're looking at an artificial separation which is the result of computing generally sucking

the separation between development, production, runtime, compile time, environments, your process, another process, the database can these be unified?

Microsoft Excel?

Was thinking more of DBOS, Smalltalk, Urbit and its forks, Erlang

The tricky thing with database configuration is setting expectations around what happens when you mutate the data in the database (does it take effect immediately, etc)

Configuration files are nice because you can edit them using tools like sed in scripts

Often you tend to see stuff like process level configuration in files, and sort of entire system level configuration/tunables in some kind of database

also, how do you get to the database? probably via some configuration. some stuff is just needed at startup before you are talking to the database

Yeah, my first reaction to the question was: where do you keep your DB connection configuration? 🙂 If you have "config" in the database, how do you track changes to it? How do you propagate those changes across environments? What about all the other configuration the environments might need beyond your application which can connect to the database? There's definitely some "configuration" which can be convenient to have in a database -- if it's the sort of thing the business team might want to change without needing developer effort: application metadata, if you will.

Environment variables exist for a good reason for example.

also, how do you get to the database?

> How do you propagate those changes across environments? I need to create an example we can talk about, so for example, you have a dev and prod s3 bucket. And the "change you want to propagate" is that the prod and dev bucket should both update to a new value. In that case, I can imagine their are two config databases, one for dev and one for prod and you have to send a transaction to both.

As with all things, I would start small, and just use a file for my configuration to. I just wander why I see people reaching for storing secrets and configuration things like AWS secrets manager and s3 before the considering re-using there existing database knowledge.

AWS secret manager is, iirc at some level backed by secure enclaves implemented by hardware, so it is in some sense a database, just built around storing stuff securely at rest, which databases are usually pretty bad at

S3 is an object store, and often has a lot more in common with filesystens than databases, and has the nice property that you can generate a URL for reading or writing via relatively common tools like curl

I'm very fond of aero + component, having the same shape for config and system map seems healthy Otherwise, there are plenty of solutions for storing configuration in a distributed / secure manner. AWS or vault for secrets, consul for service discovery or dynamic configuration, unleash for feature flags At some point Just A Text File becomes more convenient, and also tracks your code Common configuration - git submodule?

this is a familiar rabbit hole for me, and its one that led me down the dark Guix/Nix-adjacent alleys full of shifty figures with knives and un-spayed/un-neutered cats (it was good!). I do like Aero quite a bit too :^) but yea otherwise the Infra/Config-as-Code/as-Data philosopy gets very convoluted when you get to the lower levels, and the wave function does eventually need to collapse.

Obviously, we need a Clojure substitute for Guix/Nix

We use dependency injection - so something (like a DB connection string) can come from anywhere (DB, a file, environmental variable, database, API call, code etc) - the component receiving it has no idea and doesn't care. We can always change where the value comes from later purely in configuration (e.g. environmental variable -> DB) and don't have to change any code.

Things like environmental variables work when you have a small number of databases - but let's say you're creating a DB management tool and it connects to thousands/millions of databases - you may well want those connection details in a database instead of an environment variable.

In what way does configuration, e.g "database-url" distinguish itself from data kept in a database?

It doesn't matter so much where you store most of the stuff, but how you bootstrap your access to that storage. Secrets are a concern and different stages/environments also. Once your bootstrapped, you can get the rest of the config from nearly anywhere.

"Bootstrap access to storage" inevitably boils down to "someone has the master passkey locked down, preferably backed up in three different secure locations, and with a bullet-proof succession plan in case they throw up their hands in frustration and go away to raise chickens at a farm".

No one has to have access to the passkeys. (1) You can split them into small chunks and give each chunk to a different person or (2) you can have the application automatically set its own random password the first time it launches.

One approach is to put all the passkeys in the config file but encrypt them (so no one in the team can read them). Then the application just needs to fetch one decryption key from the secret store and all configuration is in one place.

on the subject of passwords and access, we have a convoluted scheme which works rather well: • vault token injected into container as a file which is cycled repeatedly • that is used to read passwords from vault, including for DBs and AWS • AWS might also require some assume-role flow

> then the application just needs to fetch one decryption key from the secret store and all configuration is in one place. Sorry, to be clear, by "master passkey" I meant this type of master key. The one that unlocks the others. This one must become Voldermort's Horcruxes.

My point about encrypted config was particularly a counter point to this. > I think the icky part of configuration isn't the configuration format itself, but the operational difficulty of managing secrets. Secrets have to be stored secretly, and shared secretly. > > By construction, any nontrivial configuration is always split across two systems (at least)...

The issue with a database for configuration, is that it's out of sync with the code. If you store config alongside your source code in git (which you can think of git as a database, and hence you ARE storing config in a DB), it will always be in-sync with the code that consumes the configuration. For example, if you deployed a broken build, but the issue is a combination of the config + code, you can just roll back to the previous version of the app and it will work. If your config was in a DB, you have to roll that back, but now your old code don't work with the new config and new code don't work with the old config, and that can get trickier. And if you treat the config as code that way, you get PR reviews (so 2 person rule), merge conflicts between members who both modified the config at the same time, etc.

I would only move config to a DB if you're trying to actually make it so you can configure the application behavior dynamically at runtime. So that updating the config should immediately reflect on the running application without having to restart it or re-deploy. And there are even specialized services just for that, almost like DBs made just for this use-case, such as: https://docs.aws.amazon.com/appconfig/latest/userguide/what-is-appconfig.html

Rupert (All Street): My point about encrypted config was particularly a counter point to this.

Depends on what you call a config. I would call a secret stored in some vault a config, because it depends on the environment, should not go with the code and is not app metadata as defined above.

Ya, realizing a well known ubiquitous terminology around this might be missing. I like to say: 1. You configure a system to operate it. This, I prefer it living in the code, unless you explicitly want the ability to reconfigure certain aspects at runtime. Static vs Dynamic configuration. 2. You customize the behavior of a system, to alter its functionality. This I prefer to make dynamic, where it can be modified at runtime, and exposed to the behavior owners, which is often the business teams, not the operator of the system (aka not the engineers). Because if you make this statically customizable, it'll require the operator to get involved to make the customization and restart the application. 3. You acquire secrets from a secure location. Secrets are kept as secrets. Some can be acquired statically, others must be acquired at runtime because they rotate. So I make a distinction between configuration, customization and secrets, and treat them as three separate things. Each one I have a different way to handle them.