Celery "Scheduler: Sending due task" logs printed as info level when expectation is debug

Question

I'm still new to Python and Celery. I've setup Celery app to send all logs to Google Cloud Logging.

It seems that part of the logs are sent as DEBUG, but some as INFO, causing me so much strain when trying to read them...

Example logging handler

LOGGING_GCP = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
    'standard': {
        'class': 'logging.Formatter',
        'format': '%(message)s | %(module)s:%(lineno)d',
    },
},
'handlers': {
    'gcp': {
        'level': 'DEBUG',
        'class': 'google.cloud.logging.handlers.ContainerEngineHandler',
        'formatter': 'standard',
    },
},
'loggers': {
    '': {
        'handlers': ['gcp'],
        'level': 'DEBUG',
        'propagate': 'False',
    }
}

}

Celery is started with --loglevel DEBUG.

Scheduler config

    CELERYD_HIJACK_ROOT_LOGGER=False,
    CELERYD_REDIRECT_STDOUTS=False,

If I check logs I below as DEBUG:

"tasks.someTaskName sent. id->26108d79-46cd-41dc-a074-925fc2dedff8"
"Task accepted: tasks.someTaskName[6781cc58-d158-45dd-a9a8-5b6c26239a79] pid:8"
"beat: Waking up now."

While these are printed as INFO:

"Received task: tasks.someTaskName[59955295-2dc1-4667-b163-2c4181658dd3] "
"Scheduler: Sending due task someTaskName (tasks.someTaskName)"
"Task tasks.someTaskName[6781cc58-d158-45dd-a9a8-5b6c26239a79] succeeded in 0.0221633310002s: None"

I'd like INFO tasks to be printed as DEBUG so that my actual application INFO logs are clearly separated.

Answer 1

You can't change how Celery creates the logs: the Celery code calls log.info(...) for logs that the developers think should be INFO, log.debug(...) for DEBUG, etc.

You can change what your application does with those logs. The loggers section in your config describes how to configure specific loggers: each key is a logger name. The logger names are usually module names, so you can specify them at any level that makes sense.

If you don't want some INFO entries at all, set the logger to WARN for that subset. This will log WARN+ for modules under celery.worker, and INFO+ for other all celery logs:

    'celery': {
        'handlers': ['gcp'],
        'level': 'INFO',
    },
    'celery.worker': {
        'handlers': ['gcp'],
        'level': 'WARN',
    }

If you really don't want celery logs mixed with your application logs, you could use a different handler to send them somewhere else:

    'celery': {
        'handlers': ['gcp-celery'],
        'level': 'DEBUG',
    }