Celery tasks from different applications in different log files

Question

I'm looking for configure Celery on my FreeBSD server and I get some issues according to log files.

My configuration:

• FreeBSD server
• 2 Django applications : app1 and app2
• Celery is daemonized and Redis
• Each application has his own Celery task

My Celery config file:

I have in /etc/default/celeryd_app1 :

# Names of nodes to start
CELERYD_NODES="worker"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/www/app1/venv/bin/celery"

# App instance to use
CELERY_APP="main"

# Where to chdir at start.
CELERYD_CHDIR="/usr/local/www/app1/src/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/app1/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/app1/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

I have exactly the same file for celeryd_app2

Django settings file with Celery settings:

CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False
CELERY_TASK_TRACK_STARTED = True
# Add a one-minute timeout to all Celery tasks.
CELERYD_TASK_SOFT_TIME_LIMIT = 60

Both settings have the same redis' port.

My issue:

When I execute a celery task for app1, I find logs from this task in app2 log file with an issue like this :

Received unregistered task of type 'app1.task.my_task_for_app1'
...
KeyError: 'app1.task.my_task_for_app1'

There is an issue in my Celery config file ? I have to set different redis port ? If yes, How I can do that ?

Thank you very much

Answer 1

I guess the problem lies in the fact that you are using the same Redis database for both applications:

CELERY_BROKER_URL = 'redis://localhost:6379'

Take a look into the guide for using Redis as a broker. Just change the database for each application, e.g.

CELERY_BROKER_URL = 'redis://localhost:6379/0'

and

CELERY_BROKER_URL = 'redis://localhost:6379/1'