Fork me on GitHub
Martynas Maciulevičius13:06:55

Hey. My friend told me that he had a problem when he was upgrading his AWS RDS DB using terraform. He said he experienced 10 minutes of downtime. I think he was upgrading between major versions. How do I write my terraform config so that this wouldn't happen?


The output of terraform plan will tell you what it plans to change. Consult with the AWS documentation to find out what the implications are for those changes.


Your friend was lucky if they weren’t doing that already - some seemingly simple changes require deleting and recreating resources in AWS, including databases!

Martynas Maciulevičius13:06:40

I think he caused prod to go down for 10 minutes. I didn't yet write a single line of terraform as I'm afraid of the options. 😄 😅


Some things in AWS require downtime - I’m not sure what those are for RDS, and it may depend on the database, but I wouldn’t be surprised if resizing is one of them. Others can’t be done directly and require deletion and recreation - for some resources, name changes are like that. Your little PR going from kabob-case to snake-case in identifiers can get a little scary!

Martynas Maciulevičius13:06:20

What does it mean to recreating resources in AWS, including databases? Does it drop the instance or does it drop the storage too? So if it will recreate a DB then will it irrecoverably delete the contents and won't restore?


Up to and including deleting data.


It’s on you to understand the terraform plan, and cross check with AWS docs.

Martynas Maciulevičius13:06:03

I think that my friend said this: "well, you could use terraform with CI-CD stuff". And in my opinion the deletion may mean that you probably wouldn't want to use it with CI-CD for more than execution where you looked up through the plan beforehand :thinking_face:


Yeah, you’d absolutely want a human in the loop for review there.


Usually the CI/CD aspect is there to separate out the roles of writing the terraform from reviewing the plan, and to keep the high-power credentials used to apply the plans out of grubby human hands.

👍 1

Updating RDS always has downtime, even without terraform. Major versions take 10-30 minutes, minor versions are quicker.