RDS Proxy rotating passwords

Question

We are using the alternating users rotation strategy to make sure our RDS passwords are rotated regularly.

Our warm (and provisioned) lambdas hang on to the current userid/password for a short period (approx. 1 hour) to reduce the number of times we need to hit Secrets Manager for a credential that only changes every X months.

The cached credential is still a perfectly valid RDS userid/password even after the new one is made "active" (explicitly by design as part of this rotation strategy), but the problem is that RDS Proxy barfs if a lambda tries to establish a connection using it because the newly updated Secret no longer contains that userid/password as the current credential. Connecting directly to the database (instead of via the proxy) works just fine.

The only way I can see around this is to change my lambdas to read the secret value /every/ time - which is a) super slow, b) has a financial impact, and c) wasteful for a password rotation that only happens every X months.

Anyone managed to find a way to beat RDS Proxy into submission in this scenario?

Answer 1

Do your normal cache as usual but wrap it in a boolean like shouldRetrieveCredentialsFromSecretsManager, then try..catch the connection handler, and if you receive a password error, flip the boolean, re-request the secret from Secrets Manager, and update the cache.

This way you're not requesting it every time, you simply evict the cache once you get an authentication error and rebuild it.

Answer 2

I agree that having some logic to retry reading from SecretsManager if the cached credentials don't work is a good policy. But if you're interested in implementing multi-user credential rotation with RDS Proxy, you can do so by having separate secrets for your different users. For RDS Proxy to allow credentials, they have to be valid in Postgres and they also have to match the username and password fields in one of the secrets that is part of the proxy's auth scheme. You can even make this work with automatic secrets rotation through SecretsManager. As a reminder, SecretsManager automatic rotation uses four steps:

1. Create the new secret
2. Change the credentials
3. Test the credentials
4. Update the secret to use the new credentials

You can maintain a separate set of secrets for RDS proxy and have your step 2 above not only update the Postgres credentials but also update the proxy-specific secret. This same strategy can be used no matter how you update the credentials. Just make sure you have a backup secret in RDS proxy that lasts as long as your desired cache time. Also keep in mind that RDS proxy takes about a minute to notice changes to either the list of secrets or to the values of secrets it uses for authentication.

If you want a little more, I wrote a blog post about this based on how I solved it at my organization. Here's the post: https://medium.com/@jberkenbilt/rotating-secrets-with-aws-rds-proxy-9b5f1c4f52e5