Asyncio: how to read stdout from subprocess?

Question

I have stuck with a pretty simple problem - I can't communicate with process' stdout. The process is a simple stopwatch, so I'd be able to start it, stop and get current time.

The code of stopwatch is:

import argparse
import time

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('start', type=int, default=0)

    start = parser.parse_args().start
    
    while True:
        print(start)
        start += 1
        time.sleep(1)

if __name__ == "__main__":
    main()

And its manager is:

import asyncio


class RobotManager:
    def __init__(self):
        self.cmd = ["python", "stopwatch.py", "10"]
        self.robot = None

    async def start(self):
        self.robot = await asyncio.create_subprocess_exec(
            *self.cmd,
            stdout=asyncio.subprocess.PIPE,
        )

    async def stop(self):
        if self.robot:
            self.robot.kill()
            stdout = await self.robot.stdout.readline()
            print(stdout)
            await self.robot.wait()
        self.robot = None


async def main():
    robot = RobotManager()
    await robot.start()
    await asyncio.sleep(3)
    await robot.stop()
    await robot.start()
    await asyncio.sleep(3)
    await robot.stop()

asyncio.run(main())

But stdout.readline returns an empty byte string every time.

When changing stdout = await self.robot.stdout.readline() to stdout, _ = await self.robot.communicate(), the result is still an empty byte string.

When adding await self.robot.stdout.readline() to the end of the RobotManager.start method, it hangs forever.

However, when removing stdout=asyncio.subprocess.PIPE and all readline calls, the subprocess prints to the terminal as expected.

How do I read from the subprocess stdout correctly?

Answer 1

In this case "proc.communicate" cannot be used; it's not suitable for the purpose, since the OP wants to interrupt a running process. The sample code in the Python docs also shows how to directly read the piped stdout in these cases, so there is in principle nothing wrong with doing that.

The main problem is that the stopwatch process is buffering the output. Try using as command: ["python", "-u", "stopwatch.py", "3"] For debugging it will also help to add some prints indicating when the robot started and ended. The following works for me:

class RobotManager:
    def __init__(self):
        self.cmd = ["python", "-u", "stopwatch.py", "3"]
        self.robot = None

    async def start(self):
        print("======Starting======")
        self.robot = await asyncio.create_subprocess_exec(
            *self.cmd,
            stdout=asyncio.subprocess.PIPE,
        )
        
    async def stop(self):
        if self.robot:            
            self.robot.kill()
            stdout = await self.robot.stdout.readline()
            while stdout:
                print(stdout)
                stdout = await self.robot.stdout.readline()            
            await self.robot.wait()
            print("======Terminated======")
        self.robot = None

Answer 2

if you don’t need to process output as it happens, but can wait for the process to end, use this:

await proc.wait()
output = await proc.stdout.read()

IMHO, you should really only use proc.communicate to send stdin to the process.