How to restart Kubernetes pod when a secret is updated in Hashicorp Vault?

Question

Have successfully implemented Vault with Kubernetes and applications running in K8s are getting their environment variables from Hashicorp vault. Everything is great! But, want to take a step forward and want to restart the pod whenever a change is made to the secret in the Vault, as of now, we have to restart the pod manually to reset environment variables whenever we make changes to Vault secret. How this can be achieved? Have heard about confd but not sure how it can be implemented!

Answer 1

Finally, just out of curiosity, came up with an idea to build an Operator to watch Vault for any change in a specific Secret Engine and restart pods of respective Deployment defined in a Custom Resource. Here is the medium blog on it with the code base - https://medium.com/@ashitacharya1/the-auto-rollout-of-kubernetes-pods-upon-change-of-hashicorp-vault-secrets-kopf-operator-670aa670def0 Might come helpful for a few.

Answer 2

Use reloader https://github.com/stakater/Reloader. We found it quite useful in our cluster. It does a rolling update hence you can change config with zero downtime too. Also if you made some errors in configmap you can easily do a rollback.

Answer 3

https://www.vaultproject.io/docs/agent/template#renewals-and-updating-secrets

If a secret or token isn't renewable or leased, Vault Agent will fetch the secret every 5 minutes. This is not configurable. Non-renewable secrets include (but not limited to) KV Version 2.

Answer 4

A couple ideas, depending on how much effort you want to put into it:

1. Just restart the pod every so often. A hacky way to do this is with a liveness probe, like this answer. Drawback is you can't use the liveness probe as a real health check without additional scripting.

2. Create an operator that polls Vault for changes and instructs Kubernetes to restart the pod when a change is detected. Not sure if Vault has an events API that you could use for that.