We're using AWS EKS in our environment.
We have 3 nodes currently, and using ASG for node group. (Two nodes are in A zone, and one node is in C zone)
I have an issue with pvc attachment problem in our deployment.
The app that has issue is for db, and we're using deployment for app because the role of db application is a collector that collects solar power plant field equipment data in real time. To minimize downtime during deployment, the deployment method is used.
- deploy method: deployment
- update method: rolling update
- replica set: 1
We have attached pvc in deployment, and the access mode is "ReadWriteOnce"
A few days ago, we deployed changed to the db app, but a problem occurred here.
Since the rolling update deploy method is used, new pod was init while previous pod is maintained. However while new pod was initializing, the multi attach error occurred.
As far as I know, "ReadWriteOnce" mode can only be accessed by one node at a time. So, I think the error occurred because the new pod was started to a different node and could not access the pvc.
To solve this error, I thought of the following method.
Change the deploy method to statulset. --> Currently, the rolling update method of deployment is being used to minimize downtime. Is it possible to minimize downtime if I use statefulset?
Use efs and change to ReadWriteMany access mode. --> I think it costs a bit more and is slower than EBS.
Set the pod to upload only to specific nodes. (node selector, node affinity,..) --> I am worried about the problems that will arise if the node instance scales down and disappears.
Use the deployment method and recreate update method. --> Because existing pods are deleted and new pods are created, downtime is expected to occur during that time.
And the strange thing here is that the app has been used since October 2023, and has been deployed several times, but this error has never occurred.
Or, unlike k8s, in EKS, when a new pod is created on another node, does EBS automatically move to that node?
I would like to ask for opinions on the best method for the above situation and why this error did not occur before.
Deployment is being done in the following way: backup the old deployment > delete the old deployment > re-deploy new deployment.
This problem has not been resolved so far. Please help me!