Daemonizing Celery--No module named 'celery' error

Question

I'm trying to daemonize Celery 4.3.0 on Ubuntu 18 following the official documentation. This is for a Django project to off-load intensive tasks.

When the server is running the celery.service should make the Celery worker available to process tasks. However, Apache2 won't even run. If I tail the Apache log I see:

[Sun Sep 29 07:42:07.621273 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[Sun Sep 29 07:42:07.621279 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[Sun Sep 29 07:42:07.621285 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[Sun Sep 29 07:42:07.621291 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
[Sun Sep 29 07:42:07.621297 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[Sun Sep 29 07:42:07.621303 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "/var/www/html/examgap/examgap/__init__.py", line 5, in <module>
[Sun Sep 29 07:42:07.621307 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]     from .celery import app as celery_app
[Sun Sep 29 07:42:07.621313 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "/var/www/html/examgap/examgap/celery.py", line 5, in <module>
[Sun Sep 29 07:42:07.621317 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]     from celery import Celery
[Sun Sep 29 07:42:07.621333 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952] ModuleNotFoundError: No module named 'celery'

Celery is definitely installed for Python3. I can run the Celery worker for my app standalone using celery -A examgap worker -l warning--just not as a daemon.

I can also import celery if I launch a Python3 shell.

I've looked at several existing SO questions like this one and think I've eliminated some possibilities. For example, the daemon user is ubuntu and this has rwx permissions for my project folder.

Still I think the problem lies with my celery.service file and the command that starts the daemon either not accessing the right folder or using the right permissions.

Here's my /etc/systemd/system/celery.service:

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=ubuntu
Group=ubuntu
EnvironmentFile=/etc/conf.d/celery
WorkingDirectory=/opt/celery
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} --pidfile=${CELERYD_PID_FILE}' ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

Thanks in advance.

Edit: Here's my /etc/conf.d/celery too:

# Name of nodes to start
# here we have a single node
CELERYD_NODES="eg1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute or relative path to the 'celery' command:
#CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
#Think I installed Celery only for user ubuntu
CELERY_BIN="/home/ubuntu/.local/bin/celery"

#CELERYD_CHDIR="/var/www/html/examgap/"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="examgap"
# or fully qualified:
# CELERY_APP="examgap.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

Edit 2: Changed the WorkingDirectory to my project directory and now the celery service is running. Yet the ModuleNotFoundError: No module named 'celery' persists.

Answer 1

[Sun Sep 29 07:42:07.621313 2019] [wsgi:error] [pid 2648:tid 140134825535232] [remote 92.4.204.209:55952]   File "/var/www/html/examgap/examgap/celery.py", line 5, in <module>

Seems like you have your own celery.py, if so you should rename it to avoid import problems

Answer 2

Here I let some links that look excellent. Just now I'm doing the same.

1. https://realpython.com/asynchronous-tasks-with-django-and-celery/ (recommended)
2. http://blog.fixter.org/hospedando-django-celery-rabbitmq-en-aws-parte-1/ (i think it's tricky way)
3. https://dev.to/idrisrampurawala/deploying-django-with-celery-and-redis-on-ubuntu-3fo6 (level pro)

I hope this work for you.