python loguru outputs error to terminal but not to log file

Question

I'm running into an issue with loguru for logging where everything works for stdout but only partially for stderr.

The problem is:

• For my Local Terminal:

  • Regular logs - do output
  • Errors - do output

• For the log file

  • Regular logs - do display
  • Errors - do not display!

I have a cron job that runs the below bash script. It essentially just activates the python env, timestamps the log file, and runs a python script. Ubuntu is the OS.

#!/bin/bash 

log_file="backend/cron_run.log"

# get the dir of this script and cd to the Winions.gg dir
CURR_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd $CURR_DIR/..

source backend/venv/bin/activate

# output the date in EST encased in brackets to the log file after 2 new lines
# output the same, except with 0 new lines if the file is empty
if [ -s $log_file ]
then
    echo -e "\n\n[$(TZ=America/Los_Angeles date)]" >> $log_file
else
    echo -e "[$(TZ=America/Los_Angeles date)]" >> $log_file
fi

python3 backend/Scripts/recap/recap.py

This is the loguru python config I have set up.

logger.remove(0)
log_format = "<green>{time:YYYY-MM-DD HH:mm:ss.SSS zz}</green> | <level>{level: <8}</level> | <yellow>Line {line: >4} ({file}):</yellow> <b>{message}</b>"
logger.add(sys.stdout, level=log_level, format=log_format, colorize=True, backtrace=True, diagnose=True)
logger.add(root_dir + '/backend/cron_run.log', rotation='2 MB', level=log_level, format=log_format, colorize=False, backtrace=True, diagnose=True)

As you can see, I've set it up with the aim of logging everything to stdout and the log file simultaneously. The problem is as I've described above: regular logging happens for my terminal and the log file, but errors only get outputted to my terminal and not the log file.

This is true whether I run the bash file, run the python file directly, or have the cron job run the bash file.

I've also tried playing around with the last line of the bash file, adding 2>&1 or other variations to output it to the log file, but I want loguru to be able to handle it for continuity and formatting reasons.

I've tried adding another sink with sys.stderr, but nothing changes. I think this is either me not understanding loguru or stderr/stdout.

Answer 1

By "error" I assume you mean usual Exception raised by Python when an unexpected problem occurs, such as ZeroDivisionError with the following code for example:

def divide():
    1 / 0  # Error!

In this case, if there is no try / except clause to catch the exception, the error will instead be handled by the default sys.excepthook(). This hook will print the error on sys.stderr before exiting the program.

However, because the logger wasn't made aware of the error, there is no way it can log it. You just feel it's partly working because your first sink and the default Python exception handler have the same destination (sys.stdout and sys.stderr being both rendered by your terminal). The cron_run.log file is not so lucky: it only receives your logs, and the Python interpreter won't redirect the error to it.

This is an unfortunate problem and Loguru has a function designed to ease logging unexpected errors like this: the @logger.catch decorator. You can use it to wrap your main() function for example:

from loguru import logger

def divide():
    1 / 0  # Error!

@logger.catch
def main():
    divide()

if __name__ == "__main__":
    main()

This is a helper that uses a try / except clause internally to capture unhandled exceptions and log them to all configured sinks.