How can I call a method on a handler created by logging.config.dictConfig in Python?

Question

I'm trying to set up a particular logging scheme for an application I'm building. For it, I want to be able to rotate logs arbitrarily on a custom condition. As such, the built-in options of rotating based on time (using TimedRotatingFileHandler) or log size (using RotatingFileHandler) are not sufficient. Both TimedRotatingFileHandler and RotatingFileHandler do however have the method doRollover which I could use to implement what I want. The problem comes from that I'm using logging.config.dictConfig to set up my log configuration, like so:

config = {
    "version": 1,
    "formatters": {
        "default_formatter": {
            "format": logging.BASIC_FORMAT,
        },
    },
    "handlers": {
        "file": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "NOTSET",
            "formatter": "default_formatter",
            "backupCount": 5,
            "filename": "log.txt",
            "encoding": "utf8",
        },
    },
    "loggers": {
        "": {
            "level": "NOTSET",
            "handlers": ["file"],
        },
    },
}
logging.config.dictConfig(config)

This way, logging.config.dictConfig is responsible for instantiating RotatingFileHandler, and so I never get the chance to retain a reference to the class instance (i.e., the object). As such, it is not clear how I could go about calling methods upon the object.

How could I go about calling a method (in my case, doRollover) on an object instantiated as a handler by logging.config.dictConfig? Alternatively, if that is not possible, how can I manually provide a handler object that I have instantiated by calling the constructor directly given this configuration?

Answer 1

This can be done by assessing the handlers property of the logger the handler is installed onto. In the code snippet above, the root logger is assigned the handler, and so we can get it like so:

file_handler_name = "file"
root_logger = logging.getLogger()

file_handler = next(
    handler for handler in root_logger.handlers if handler.name == file_handler_name
)
file_handler.doRollover()  # or whatever other method you want to call

This works because in the question's example, the handler is named "file", and in here we essentially iterate through root_logger.handlers until we find one where the name property is equal to "file". The handler returned by this iteration is then the object that was instantiated by logging.config.dictConfig.

If using next isn't to your liking, the equivalent with a for-loop would be this:

file_handler_name = "file"
root_logger = logging.getLogger()

for handler in root_logger.handlers:
    if handler.name == "file":
        file_handler = handler
        break

file_handler.doRollover()  # or whatever other method you want to call

Answer 2

The logic for if and when to perform a rollover is encoded in BaseRotatingHandler.emit:

 def emit(self, record):
    try:
        if self.shouldRollover(record):
            self.doRollover()
        logging.FileHandler.emit(self, record)
    except Exception:
        self.handleError(record)

You can define a custom subclass of BaseRotatingHandler or RotatingFileHandler, as appropriate, whose __init__ method can take additional arguments provided by the configuration dictionary and whose shouldRollover can use those arguments to implement your rollover logic.

For example,

class NewbytesHandler(RotatingFileHandler):
    def __init__(self, *, foo, **kwargs):
        super().__init__(**kwargs)
        self.foo = foo

    def shouldRollover(self, record):
        if <some logic involving self.foo>:
            return True
        else:
            return False

and in your configuration,

config = {
    ...
    "handlers": {
        "file": {
            "class": "NewbytesHandler",
            "level": "NOTSET",
            "formatter": "default_formatter",
            "foo": 7,
            "filename": "log.txt",
            "encoding": "utf8",
        },
    },
    ...
}
logging.config.dictConfig(config)