Reading STDERR and STDOUT piped from subprocess.Popen

Question

I'm writing a class that extends Popen, and (at this stage of development) I want it to write the contents of Popen.stderr and Popen.stdout to a log file in real time. At a later stage, I will want to examine the contents, but right now, I'm stuck at just reading them.

(In case it matters, I'm running Windows 10. But I'm looking for a solution that is cross platform.)

I'm reading both with the following code that runs in a thread.

while self.isRunning():
    output = self.stdout.readline().rstrip()
    if  len( output ) > 0 :
        self.processOutput( output )
          
    if self.errHandling == PIPE:  
        error = self.stderr.readline().rstrip()
        if  len( error ) > 0 :
            self.processError( error )

(processError and processOutput print to the screen and append to the logfiles.)

When the process starts, it generates several lines of STDERR, and then starts spitting out one to three lines to STDOUT per second. At least, it's supposed to. What happens instead is that when the errors stop coming, the loop above just stops at "error = self.stderr.readline().rstrip()" and waits for more errors. The process continues to run (in my test, it runs for hours), and when it finishes it generates a few more lines of STDERR, which are captured-- but the information for STDOUT is completely lost.

If I route STDERR to STDOUT instead of PIPE, it works with no problems (but I am unable to distinguish between STDOUT and STDERR.)

If I put the STDERR processor in its own thread, the entire process just halts once the errors stop coming.

It seems like I need to find a way to read from STDERR that returns blank if there is no data, which I THOUGHT "error = self.stderr.readline().rstrip()" did.

Answer 1

Okay, I am completely stumped, but it's working. I put the two-threads solution (which was actually where I started) back in so that I could just copy-paste to demonstrate to the commenter above, but.... now it's working.