Celery doesn't detect and register tasks in django python

Question

I'm facing a problem where I can't set up periodic tasks in my application in django framework using Celery.

I've literally tried everything i could find adressing this issue on the internet, transfered the code from my MacOS laptop to the PC on Windows, but nothing seems to help... I used the documentation from celery to replicate every step for the periodic tasks to work, tried using other versions of Celery, Django-Celery-beat, django-celery-results, but nothing helped, so I don't know which part of code I'm wrong at.

Would very much appreciate any help!

My apps packages versions:

---

• celery 5.3.6

• Django 4.0.10

• django-celery-beat 2.5.0

• django-celery-results 2.5.1

• django-debug-toolbar 4.2.0

• honcho 1.1.0

• mysqlclient 2.2.1

• oauthlib 3.2.2

• Pillow 10.1.0

• pip 23.3.2

• redis 5.0.1

• Python 3.8.8

My django project structure:

django-project
├── .vscode
│   └── settings.json
├── myapp
│   ├── admin.py
│   ├── apps.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   │      
│   ├── models.py
│   ├── signals.py
│   ├── tasks.py
│   ├── templates
│   │   ├── base.html
│   │   ├── home.html
│   │   ├── login.html
│   │   └── skin_list.html
│   ├── tests.py
│   ├── urls.py
│   ├── views.py
│   ├── __init__.py
│   └── __pycache__
├── manage.py
├── django-project
│   ├── asgi.py
│   ├── celeryapp.py
│   ├── config.py
│   ├── pyenv.cfg
│   ├── settings.py
│   ├── urls.py

I didn't include some files to make project structure visually readable.

Settings.py:

"""
Django settings for Marketplace_backend project.

Generated by 'django-admin startproject' using Django 4.2.7.

For more information on this file, see
https://docs.djangoproject.com/en/4.2/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/
"""

from pathlib import Path
import os

from django-project.config import * 

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


Debug=True 

ALLOWED_HOSTS = []


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'myapp', 
    'debug_toolbar',
    'social_django',
    'django_celery_beat',
    "django_celery_results",
]


MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    "debug_toolbar.middleware.DebugToolbarMiddleware",
    'social_django.middleware.SocialAuthExceptionMiddleware',
]

ROOT_URLCONF = 'Marketplace_backend.urls'


# Celery Configuration
# CELERY_BROKER_URL = 'redis://localhost:6379'
# CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Moscow'


from celery.schedules import crontab


# CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

CELERY_BEAT_SCHEDULE = {
    'test': {
        'task': 'myapp.tasks.test',
        'schedule': crontab(minute='*/1'),
    },
}

# CELERY_INCLUDE = ('myapp.tasks',)
# CELERY_IMPORTS = ('myapp.tasks',)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'social_django.context_processors.backends',
                'social_django.context_processors.login_redirect',
                'myapp.context_processors.user_social_data',
            ],
        },
    },
]


INTERNAL_IPS = [
    # ...
    "127.0.0.1",
    # ...
]


WSGI_APPLICATION = 'django-project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'app_database',
        'USER': 'root',
        'PASSWORD': '',
        'HOST': '127.0.0.1',  
        'PORT': '3306',      
    }
}

celeryapp.py:

from celery import Celery
from celery.schedules import crontab
import os
from django.conf import settings





app = Celery('django-project')


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django-project.settings')


app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

init.py in django-project inner folder:

from __future__ import absolute_import, unicode_literals
from .celeryapp import app as celery_app

__all__ = ('celery_app',)

# Tried this line - didn't help.
from myapp.tasks import test

tasks.py in myapp:

# Someone suggested these two lines (2,3) - didn't help
import django
django.setup()


from celery import shared_task
from .models import *
from .views import *


@shared_task(name='test')
def test():
    return 'test'
    

I start everything via honcho Procfile:

web: python manage.py runserver
celery: celery -A django-project worker --pool=solo -l info
beat: celery -A django-project beat -l info

When I type honcho start (start django server, celery worker, celery beat), I get this response:

(django-project) (base) yanik@MacBook-Air django-project % honcho start
17:24:48 system   | web.1 started (pid=76511)
17:24:48 system   | celery.1 started (pid=76512)
17:24:48 system   | beat.1 started (pid=76513)
17:24:48 beat.1   | [2024-01-04 17:24:48,733: INFO/MainProcess] beat: Starting...
17:24:48 celery.1 |  
17:24:48 celery.1 |  -------------- [email protected] v5.3.6 (emerald-rush)
17:24:48 celery.1 | --- * ----- 
17:24:48 celery.1 | -- *** ---- macOS-10.16-x86_64-i386-64bit 2024-01-04 17:24:48
17:24:48 celery.1 | - *** --- * --- 
17:24:48 celery.1 | -  ---------- [config]
17:24:48 celery.1 | -  ---------- .> app:         django-project:0x7f7a4603ca30
17:24:48 celery.1 | -  ---------- .> transport:   redis://127.0.0.1:6379//
17:24:48 celery.1 | -  ---------- .> results:     redis://127.0.0.1:6379/
17:24:48 celery.1 | - *** --- * --- .> concurrency: 8 (prefork)
17:24:48 celery.1 | -- *** ---- .> task events: ON
17:24:48 celery.1 | --- * ----- 
17:24:48 celery.1 |  -------------- [queues]
17:24:48 celery.1 |                 .> celery           exchange=celery(direct) key=celery
17:24:48 celery.1 |                 
17:24:48 celery.1 | 
17:24:48 celery.1 | [tasks]
17:24:48 celery.1 |   . django-project.celeryapp.debug_task
17:24:48 celery.1 | 
17:24:49 celery.1 | [2024-01-04 17:24:49,145: WARNING/MainProcess] /Users/yanik/.local/share/virtualenvs/django-project-9ZCnhA-p/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
17:24:49 celery.1 | whether broker connection retries are made during startup in Celery 6.0 and above.
17:24:49 celery.1 | If you wish to retain the existing behavior for retrying connections on startup,
17:24:49 celery.1 | you should set broker_connection_retry_on_startup to True.
17:24:49 celery.1 |   warnings.warn(
17:24:49 celery.1 | 
17:24:49 celery.1 | [2024-01-04 17:24:49,157: INFO/MainProcess] Connected to redis://127.0.0.1:6379//
17:24:49 celery.1 | [2024-01-04 17:24:49,158: WARNING/MainProcess] /Users/yanik/.local/share/virtualenvs/Marketplace_backend-9ZCnhA-p/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
17:24:49 celery.1 | whether broker connection retries are made during startup in Celery 6.0 and above.
17:24:49 celery.1 | If you wish to retain the existing behavior for retrying connections on startup,
17:24:49 celery.1 | you should set broker_connection_retry_on_startup to True.
17:24:49 celery.1 |   warnings.warn(
17:24:49 celery.1 | 
17:24:49 celery.1 | [2024-01-04 17:24:49,161: INFO/MainProcess] mingle: searching for neighbors
17:24:49 web.1    | Watching for file changes with StatReloader
17:24:50 celery.1 | [2024-01-04 17:24:50,172: INFO/MainProcess] mingle: all alone
17:24:50 celery.1 | [2024-01-04 17:24:50,189: INFO/MainProcess] [email protected] ready.

I get this error, when celery beat tries to execute periodic task:

20:04:00 beat.1   | [2024-01-01 20:04:00,000: INFO/MainProcess] Scheduler: Sending due task  
20:04:00 beat.1   | test (myapp.tasks.test)
20:04:00 celery.1 | [2024-01-01 20:04:00,003: ERROR/MainProcess] Received unregistered task of type 'myapp.tasks.test'.
20:04:00 celery.1 | The message has been ignored and discarded.
20:04:00 celery.1 | 
20:04:00 celery.1 | Did you remember to import the module containing this task?
20:04:00 celery.1 | Or maybe you're using relative imports?
20:04:00 celery.1 | 
20:04:00 celery.1 | Please see
20:04:00 celery.1 | https://docs.celeryq.dev/en/latest/internals/protocol.html
20:04:00 celery.1 | for more information.
20:04:00 celery.1 | 
20:04:00 celery.1 | The full contents of the message body was:
20:04:00 celery.1 | b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)
20:04:00 celery.1 | 
20:04:00 celery.1 | The full contents of the message headers:
20:04:00 celery.1 | {'lang': 'py', 'task': 'myapp.tasks.test', 'id': '332beab6-a3de-4e4a-84f3-59d6041a1b47', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '332beab6-a3de-4e4a-84f3-59d6041a1b47', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen24368@DESKTOP-KNKGMAO', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}}
20:04:00 celery.1 | 
20:04:00 celery.1 | The delivery info for this task is:
20:04:00 celery.1 | {'exchange': '', 'routing_key': 'celery'}
20:04:00 celery.1 | Traceback (most recent call last):
20:04:00 celery.1 |   File "c:\users\yantb\desktop\django-project\django-project\lib\site-packages\celery\worker\consumer\consumer.py", line 658, in on_task_received
20:04:00 celery.1 |     strategy = strategies[type_]
20:04:00 celery.1 | KeyError: 'myapp.tasks.test'

Answer 1

Can't actually explain the technical aspects of how does the celery cooperate with django, but it seems that it didn't detect django's settings module. That's why celery didn't detect tasks. I made the changes suggested here: https://stackoverflow.com/a/49538946/23181652

Basically, I modified celery worker and celery beat launch commands, so the new Procfile for Honcho is:

web: python manage.py runserver
celery: DJANGO_SETTINGS_MODULE=django-project.settings celery -A django-project worker -l INFO 
beat: DJANGO_SETTINGS_MODULE=django-project.settings celery -A django-project beat -l info

And I removed shared_task name from tasks.py:

@shared_task
def test():
    return 'test'

And it worked! Thanks @wiaterb for the inspiration for the analysis!