How can I ensure synchronized incrementation of two counters in a Celery async task

44 Views Asked by At

I'm encountering a scenario where a single endpoint is receiving multiple concurrent requests for image uploads. Upon successful upload, two counters need to be incremented synchronously. Initially, I tried synchronously incrementing the counters, but due to race conditions, the results were incorrect. Consequently, I implemented an asynchronous Celery task to handle this. However, while one counter stays in sync, the other does not. I'm puzzled why only one counter is affected if both are subjected to the same conditions. I am using Postgres as my database.

To address the issue of asynchronous counter incrementation, one potential solution is to remove the counters altogether and instead track the number of uploaded images or use table locking using select_for_update. However, it's crucial to understand the root cause of the asynchronous behavior to prevent similar issues in the future.

This is the piece of code that is doing the increment

@shared_task
def increment_album_and_event_image_count(album_id):
    try:
        with transaction.atomic():
            album = Album.objects.get(id=album_id)
            album.image_count_in_album = F("image_count_in_album") + 1
            album.save()
            album.event.number_of_images_across_albums = (
                F("number_of_images_across_albums") + 1
            )
            album.event.save()
        return True
    except Exception as e:
        logger.error(f"Error incrementing image count: {e}, album: {album}")
        return False

Here the number_of_images_across_albums has correct information while image_count_in_album does not.

Here are the model details:

class Album(TimeStampedModel):
    image_count_in_album = models.IntegerField(default=0)
    event = models.ForeignKey("Event", on_delete=models.CASCADE)
class Event(TimeStampedModel):
    number_of_images_across_albums = models.IntegerField(default=0)

0

There are 0 best solutions below