How do I find coverage for subprocesses under test?

Question

Consider the following simple program:

# thingy.py
import sys

print("ready")

for line in sys.stdin:
    print(line, end='')

If I want to unit-test the program, I can stub out the side-effecting functions ready and print easily enough.

However, if I want to perform an end-to-end test, I want to run the real program and test real responses to real inputs.

I can test it easily enough:

# test_thingy.py
import pytest
import subprocess
import sys    

@pytest.fixture
def thingy_instance(): 
    proc = subprocess.Popen(
            [sys.executable, "-um", "thingy"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            encoding="utf-8")

    for line in proc.stdout:
        if "ready" in line:
            break
    yield proc
    proc.terminate()
    proc.wait()

def test_thingy(thingy_instance):
    stdout, stderr = thingy_instance.communicate("Hello!")
    assert stdout == "Hello!"

# more tests

However, if I try to generate coverage for this, it doesn't work:

$ coverage run --source . -m pytest
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/user/playground-local/python/coverage-test-stuff
plugins: Faker-16.6.1, smtpdfix-0.4.0, quarantine-2.0.0, cov-3.0.0, mock-3.8.2, docker-tools-3.1.3, postgresql-4.1.1
collected 1 item                                                               

test_thingy.py .                                                         [100%]

============================== 1 passed in 0.09s ===============================
$ coverage report
Name             Stmts   Miss  Cover
------------------------------------
test_thingy.py      15      0   100%
thingy.py            4      4     0%
------------------------------------
TOTAL               19      4    79%

Running coverage run -m thingy generates coverage.

Running coverage run --source . -m pytest doesn't generate coverage.

Replacing the Popen command with a coverage run -m command doesn't generate coverage.

Answer 1

The answer is to use the -p flag for coverage run, then run coverage combine after the fact.

From the documentation:

  -p, --parallel-mode   Append the machine name, process id and random number
                        to the data file name to simplify collecting data from
                        many processes.

To get full coverage, replace the Popen python command with the coverage command, passing -p:

@pytest.fixture
def thingy_instance(): 
    proc = subprocess.Popen(
            [sys.executable, "-um", "coverage", "run", "-pm", "thingy"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            encoding="utf-8")

    for line in proc.stdout:
        if "ready" in line:
            break
    yield proc
    proc.terminate()
    proc.wait()

And execute the top-level coverage command with -p:

$ coverage run --source . -pm pytest
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/user/playground-local/python/coverage-test-stuff
plugins: Faker-16.6.1, smtpdfix-0.4.0, quarantine-2.0.0, cov-3.0.0, mock-3.8.2, docker-tools-3.1.3, postgresql-4.1.1
collected 1 item                                                               

test_thingy.py .                                                         [100%]

============================== 1 passed in 0.16s ===============================

Then run coverage combine:

$ coverage combine
Combined data file .coverage.my-host.1383878.098470
Combined data file .coverage.my-host.1383874.892749

Finally, the coverage report (as well as coverage generation commands) will include all sub-process runs:

$ coverage report
Name             Stmts   Miss  Cover
------------------------------------
test_thingy.py      15      0   100%
thingy.py            4      0   100%
------------------------------------
TOTAL               19      0   100%