We are using
django-celery==3.1.10
celery==3.1.20
python 2.7.13
Broker - RabbitMQ
We have written a CustomDataBaseScheduler to schedule task which reads entries from a MySQL Table(Django Model) and schedules the task perfectly on given time (mysql Table column specifies the time). We are running CeleryBeat Process as init script.
Our Scheduler Model has approx 3000 entries, most of them are getting scheduled every 5 mins and some are every 15 mins and few is hourly. But out of them some tasks are getting skipped i.e doesn't get schedule on time. This behaviour is randomly and happens to any of the tasks.
While digging on beat logs, we found an MySQL exception on the logs,
Traceback (most recent call last):
File "/opt/DataMonster/datamonster/db_monster/scheduler.py", line 202, in schedule_changed
transaction.commit()
File "/opt/python-2.7.11/lib/python2.7/site-packages/django/db/transaction.py", line 154, in commit
get_connection(using).commit()
File "/opt/python-2.7.11/lib/python2.7/site-packages/django/db/backends/__init__.py", line 175, in commit
self.validate_no_atomic_block()
File "/opt/python-2.7.11/lib/python2.7/site- packages/django/db/backends/__init__.py", line 381, in validate_no_atomic_block
"This is forbidden when an 'atomic' block is active.")
Have Checked the error across multiple sites and it shows it realted to isolation level. We didn't found any other exception in BEAT logs.
The isolation method used in MySQL is READ-UNCOMMITED
Need help in digging this issue.