redirecting python logging messages with streams

Question

I want to redirect logging messages to some handling method (e.g. in order so save all messages in a queue). Currently I'm trying to make use of logging.StreamHandler in order to write to a StringIO and then read it somewhere else. In my case this might be a thread continuously reading from the stream but it could also be a callback method being called on every log entry.

import threading
import time
import sys
import logging
from StringIO import StringIO

# this thread shall read from a stream 
# continuously and 
def tread_fn( stream ):
    while not stream.eof():       <=== this is not valid but my current approach
        l = stream.readline()
        do_something( l )

stream = StringIO()

handler = logging.StreamHandler(stream)
log = logging.getLogger()
log.setLevel( logging.INFO )

# replace all log handlers
for handler in log.handlers: 
    log.removeHandler(handler)
log.addHandler(handler)

thread = threading.Thread(target = tread_fn, args=[stream])
thread.start()

for i in range(3):        
    time.sleep(1)
    log.error("test")          <=== should be handled line by line

I feel like I have overlooked the very obvious and simple best practice but I'm struggling for a while now :) Maybe I don't need streams at all but currently I'm even failing to write to a stream and reading from it somewhere else.. So in short my questions are:

• how is the main goal achieved the python way?

• how do I write strings to a stream and continuously read from it in another thread?

Answer 1

You've asked two questions in one - they should be separate questions. Your main goal can be achieved using e.g. a QueueHandler, available in Python 3.2 and later but also available for earlier Python versions through the logutils project.