CeleryBeat suddenly skips scheduling few tasks while using Custom DatabaseScheduler

91 Views Asked by At

We are using

django-celery==3.1.10
celery==3.1.20
python 2.7.13
Broker - RabbitMQ

We have written a CustomDataBaseScheduler to schedule task which reads entries from a MySQL Table(Django Model) and schedules the task perfectly on given time (mysql Table column specifies the time). We are running CeleryBeat Process as init script.

Our Scheduler Model has approx 3000 entries, most of them are getting scheduled every 5 mins and some are every 15 mins and few is hourly. But out of them some tasks are getting skipped i.e doesn't get schedule on time. This behaviour is randomly and happens to any of the tasks.

While digging on beat logs, we found an MySQL exception on the logs,

Traceback (most recent call last):
  File "/opt/DataMonster/datamonster/db_monster/scheduler.py", line 202, in schedule_changed
  transaction.commit()
  File "/opt/python-2.7.11/lib/python2.7/site-packages/django/db/transaction.py", line 154, in commit
  get_connection(using).commit()
  File "/opt/python-2.7.11/lib/python2.7/site-packages/django/db/backends/__init__.py", line 175, in commit
  self.validate_no_atomic_block()
  File "/opt/python-2.7.11/lib/python2.7/site- packages/django/db/backends/__init__.py", line 381, in validate_no_atomic_block
 "This is forbidden when an 'atomic' block is active.")

Have Checked the error across multiple sites and it shows it realted to isolation level. We didn't found any other exception in BEAT logs.

The isolation method used in MySQL is READ-UNCOMMITED

Need help in digging this issue.

0

There are 0 best solutions below