Background service in multiple pods in kubernetes

Question

I have a Background service microservice in dot net and I am deploying it to Kubernetes as a container. If I have two pods then each pod will perform the same database operation at the same time, how can I avoid this? Please help

I am expecting, if my one pod is accessing a database and fetching few records, then my another pod should not fetch the same records, if they fetch the same records, then they would perform same operation twice in the different pods

Answer 1

I see three solutions :

• Run only one instance.
• Configure worker so that each record is processed by a given worker. Pass the environment variable "rank=0" to the first worker and "rank=1" to the second worker. You will need two deployments of one instance to be able to do that. Both workers get all the records but only process the record which id modulo 2 equals rank. If you don't want the number of instances hard-coded, you can pass "rank=0" and "total=2", and then the worker only process record with id modulo total equals rank. Good : it balances the processing between workers. Bad : if you set two instances in a deployment, you still have the problem.
• Implement a kind of a lock in the database. Create a lock table with three columns : a lock name, a worker id and a date-time. When worker "1234" starts ("1234" is the pod id), it tries every 10 seconds to acquire the lock "database_lock" with this request : UPDATE lock SET worker_id="1234", date_time=CURRENT_TIMESTAMP() WHERE lock="database_lock" AND (worker_id="1234" OR date_time < CURRENT_TIMESTAMP() - 30000);. If the row can be updated, the worker has successfully "acquired" the lock and can process the database. If not, it remains idle. In both cases, it will try to acquire the lock again in 10 seconds. The lock can be acquired by a worker if it already has the lock, or it the lock has been acquired more than 30 seconds ago. So a worker that acquires a lock will normally keeps it until it its death.

I suggest the first solution (https://en.wikipedia.org/wiki/KISS_principle).