Generating/Writing Celery Logs into a file programmatically

Question

I am trying to run a simple asynchronous task using celery and rabbitmq as it's message broker. I want to generate and write the celery logs that show up in the terminal into a file programmatically.

I have a task file where I have defined my task, an application file, from where I am calling that task to be executed and also, I have my logging configured and imported for all the files. I am trying to import the logging object that I have created in my logging configuration to my celery Task, but for some reason, it doesn't generate logs. I have also tried -f LOGFILE, --logfile=LOGFILE command from the console which generates a logfile upon pushing but i want to generate it programmatically just the way regular logging happens. I have used this command to run my celery which generates the logs in the console from where i run it-- celery -A taskhandler.celery_app worker -l info -E --concurrency=1. Celery is not being able to read and process my logging object for the project and generate logs to the path I am giving. I have also tried these: https://www.distributedpython.com/2018/08/28/celery-logging/ http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging

But I have not reached to any solution.

import time
from celery.utils.log import get_task_logger
from log import Logger
from taskhandler.celery_app import celery
#logger = get_task_logger(__name__)
logger = Logger.get_logger()


@celery.task(task_ignore_result=False, track_started=True)
def train_global_trepan_task(count):
    factor = 1

    for i in range(1, count + 1):
        factor = factor * i
        time.sleep(5)
        logger.info("Task id {} factorial of {} is {}".format(train_global_trepan_task.request.id, i, factor))
    return factor

Below is the logging setting

import logging
from datetime import date
from configuration import get_config
import sys
sys.path.append("..")
class Logger:
    _logger = None

    @classmethod
    def get_logger(cls):
        name = 'logs//' + date.today().strftime("%b-%d-%Y") + '.log'
        if not cls._logger or name != cls._logger.__dict__['name']:
            logging.basicConfig(filename=name,
                                filemode='a',
                                format='%(asctime)s [%(filename)s - %(lineno)s] -%(funcName)s - %(levelname)s - %('
                                       'message)s')
            cls._logger = logging.getLogger(name)
            log_level = {
                'debug': logging.DEBUG,
                'info': logging.INFO,
                'error': logging.ERROR,
                'warning': logging.WARNING,
                'critical': logging.CRITICAL,
            }[get_config('logger', 'level').lower()]
            cls._logger.setLevel(log_level)
        return cls._logger

Expected results: Celery logs should have got generated in my log folder, the path given to it. Actual Results: Celery logs do not get generated in my log file programmatically, it gets generated in a file only when we give the celery log command in our console, and that too they are inconsistent.

Answer 1

I'd suggest to use the after_setup_logger celery decorator where you can register your logger(s) during celery startup. Then, your logger will be available inside your celery tasks.

from celery.signals import after_setup_logger

@after_setup_logger.connect
def setup_loggers(logger, *args, **kwargs):
    import logging

    fh = logging.FileHandler('/path/to/file')
    logger = logging.getLogger("test-logger")
    logger.addHandler(fh)

Answer 2

The celery signal @after_setup_logger.connect redirects celery logging to the file which gets generated when the celery starts, so if you want the information you have to look into the old directory or file and extract task details. Without restarting the celery it is not possible to get fresh task details in a new file.