Fork me on GitHub
#aws
<
2021-10-08
>
jumar09:10:58

Any recommendations on config params / secrets management on AWS? Do you use Secrets Manager or Parameter Store or both? I'd like to store the configuration in a single place. For example our beanstalk app has 50+ config properties which are right now only managed via Beanstalk config. Lots of them are secrets (like api keys) but a bunch of them is also something else. So far I'm leaning towards Parameter Store but some of the features of Secrets Manager are compelling - like seamless secrets rotation.

jumar09:10:54

I also found this: https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html Sounds like it could be a good idea to actually combine them.

Max13:10:18

I’ve tried both and I prefer https://github.com/mozilla/sops with the KMS encryption option. Config should deploy on the same lifecycle as code, if you want to change config live then you really want feature flags instead. As for my experiences with Secrets Manager and Param Store, I found they did more harm than good. This culminated in an incident where I accidentally deleted some critical and irreplaceable configs. After that we switched to a method with version control and code reviews.

lukasz14:10:50

ECS + SecretsManager is pretty great. Or alternatively - you can use a tool like Doppler or EnvKey and only use SecretsManager to store the individual service key used to obtain the secrets - reduces the blast radius

viesti20:10:11

I've been using SOPS too, for keeping secrets from integrations in a vault in version control. But if there are secrets that are internal to a system, like say a RDS master user + app user password, those could be generated inside the infrastructure, either by Terraform or by the Secrets Manager lifecycle (have to probably call rotation after initial creation). Vaults are nice for versioning though.

AJ Jaro11:10:43

We just switched from Beanstalk config to the Parameter Store and it has worked fine so far. I, however, don't have experience using the secrets manager

jumar04:10:44

@UGMAVSMUM how did you do that? Do you have the configuration values version controlled and updated automatically in the parameter store?

jumar10:10:33

@U01EB0V3H39 and @U06QSF3BK thanks a lot for the valueable info. With SOPS + KMS, are you still able to reproduce/fetch configuration as it was at any point in time in the past? What happens when you rotate keys and the config files get re-encrypted? Also, how do you integrate SOPS with your apps? Do you just run sops and store its output somewhere in the file a config file read by the application or you use something more sophisticated?

jumar10:10:55

I'm wondering if I could/should just invoke sops from the app process when it starts and read the configuration dynamically without storing the plaintext anywhere....

jumar10:10:54

So in my case, I could package the encrypted file within the Beanstalk app archive and when the app starts it would use sops to get the plaintext version of the config. It would be great to integrate this somehow with a fine clojure config library like aero, omniconf, etc.

AJ Jaro12:10:17

@U06BE1L6T They are not currently version controlled so that's something we still need to figure out, yep.

Max13:10:43

My sops files are checked into git. I haven’t had to think much about key rotations, so that would be a flaw in the plan. Maybe there’s a way to archive old kms keys rather than delete them when rotating? We use both aero and sops: the app startup command is actually a sops command that decrypts the secrets and injects them into the app process as environment variables. Then aero brings in the env vars. This is generally good practice: your config will likely be more complicated than the values you need to inject into it, and injecting config via env vars is in line with 12 factor principles (in addition to just being a good idea in general)

Max13:10:19

I would avoid integrating sops into my app code, that’s not really its concern. Using env vars means I have options for different deployment modes, eg local dev. No sops needed there!

jumar18:10:33

Thanks, Max, this has been very helpful! Environment variables have their own weaknesses (like being exposed to sub-processes) but that's another discussion and I agree it's probably the most common approach (it's what we use right now anyway). And I actually see the value in using sops in local/dev environment too - we have some shared stuff in dev environment for which we share secrets (like GitHub OAuth app keys) and need to distribute those keys to devs.