`pytest` and `yield`-based tests

Question

I am trying to migrate a bunch of tests from nose to pytest and I am having trouble migrating one test that validates a whole process.

I have dumbed it down to represent my problem:

def is_equal(a, b):
    assert a == b


def inner():
    yield is_equal, 2, 2
    yield is_equal, 3, 3


def test_simple():
    yield is_equal, 0, 0
    yield is_equal, 1, 1
    for test in inner():
        yield test
    yield is_equal, 4, 4
    yield is_equal, 5, 5


def test_complex():
    integers = list()


    def update_integers():
        integers.extend([0, 1, 2, 3, 4, 5])

    yield update_integers

    for x in integers:
        yield is_equal, x, x

test_simple runs fine between nose and pytest, but test_complex only runs the initial update_integers test:

~/projects/testbox$ nosetests -v
test_nose_tests.test_simple(0, 0) ... ok
test_nose_tests.test_simple(1, 1) ... ok
test_nose_tests.test_simple(2, 2) ... ok
test_nose_tests.test_simple(3, 3) ... ok
test_nose_tests.test_simple(4, 4) ... ok
test_nose_tests.test_simple(5, 5) ... ok
test_nose_tests.test_complex ... ok
test_nose_tests.test_complex(0, 0) ... ok
test_nose_tests.test_complex(1, 1) ... ok
test_nose_tests.test_complex(2, 2) ... ok
test_nose_tests.test_complex(3, 3) ... ok
test_nose_tests.test_complex(4, 4) ... ok
test_nose_tests.test_complex(5, 5) ... ok

----------------------------------------------------------------------
Ran 13 tests in 0.004s


~/projects/testbox$ pytest -v
====================================================================     test session starts         =====================================================================
platform linux2 -- Python 2.7.12, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/local/ANT/cladam/projects/testbox, inifile: 
collected 7 items 

tests/test_nose_tests.py::test_simple::[0] PASSED
tests/test_nose_tests.py::test_simple::[1] PASSED
tests/test_nose_tests.py::test_simple::[2] PASSED
tests/test_nose_tests.py::test_simple::[3] PASSED
tests/test_nose_tests.py::test_simple::[4] PASSED
tests/test_nose_tests.py::test_simple::[5] PASSED
tests/test_nose_tests.py::test_complex::[0] PASSED

=================================================================== pytest-warning summary     ===================================================================
WC1 /home/local/ANT/cladam/projects/testbox/tests/test_nose_tests.py yield tests are deprecated, and scheduled to be removed in pytest 4.0
....
======================================================== 7 passed, 7     pytest-warnings in 0.01 seconds =========================================================

I am assuming that this is because at collection time the integers list is empty and this it doesn't then collect the 6 additional yields.

Is there any way that I can replicate this test structure in pytest? via pytest_generate_tests?

This test is representative of a larger sequence of events to build an object up and operate on it, and test at each stage of the process.

1. Model something
2. validate some model properties
3. create and output file based on model
4. diff against a known output to see if there are changes.

Thanks in advance

Answer 1

As the output of your tests suggests, yield-based tests are deprecated:

WC1 /home/local/ANT/cladam/projects/testbox/tests/test_nose_tests.py yield tests are deprecated, and scheduled to be removed in pytest 4.0

I suggest that you make use of the decorator pytest.parametrize instead. You can check more about it at:

• https://docs.pytest.org/en/latest/how-to/parametrize.html
• https://docs.pytest.org/en/latest/example/parametrize.html

From your example, I would create something like this for the tests:

import pytest


def is_equal(a, b):
    return a == b


class TestComplexScenario:
    @pytest.mark.parametrize("my_integer", [0, 1, 2])
    def test_complex(self, my_integer):
        assert is_equal(my_integer, my_integer)

Here is a sample of the output:

test_complex.py::TestComplexScenario::test_complex[0] PASSED
test_complex.py::TestComplexScenario::test_complex[1] PASSED
test_complex.py::TestComplexScenario::test_complex[2] PASSED

You can find some more examples about parametrize at: http://layer0.authentise.com/pytest-and-parametrization.html

You can also make permutations for your test inputs, check an example at: parameterized test with cartesian product of arguments in pytest

Answer 2

The issue is that pytest collects all the tests prior to running any of them, and so within test_complex the update_integers function is not called until after the collection process ends.

AYou can get the tests to run by moving the is_generator check from the collection stage to the test run stage by placing the following in conftest.py. Unfortunately, the hooks do not allow pytest_runtest_protocol to operate as a generator, so the entire content of _pytest.main.pytest_runtestloop for pytest-3.2.1 were copied and modified.

import pytest
from _pytest.compat import is_generator
def pytest_pycollect_makeitem(collector, name, obj):
    """
    Override the collector so that generators are saved as functions
    to be run during the test phase rather than the collection phase.
    """
    if collector.istestfunction(obj, name) and is_generator(obj):
        return [pytest.Function(name, collector, args=(), callobj=obj)]

def pytest_runtestloop(session):
    """
    Copy of _pytest.main.pytest_runtestloop with the session iteration
    modified to perform a subitem iteration.
    """
    if (session.testsfailed and
            not session.config.option.continue_on_collection_errors):
        raise session.Interrupted(
            "%d errors during collection" % session.testsfailed)

    if session.config.option.collectonly:
        return True

    for i, item in enumerate(session.items):
        nextitem = session.items[i + 1] if i + 1 < len(session.items) else None
        # The new functionality is here: treat all items as if they
        # might have sub-items, and run through them one by one.

        for subitem in get_subitems(item):
            subitem.config.hook.pytest_runtest_protocol(item=subitem, nextitem=nextitem)
            if getattr(session, "shouldfail", False):
                raise session.Failed(session.shouldfail)
            if session.shouldstop:
                raise session.Interrupted(session.shouldstop)
    return True

def get_subitems(item):
    """
    Return a sequence of subitems for the given item.  If the item is
    not a generator, then just yield the item itself as the sequence.
    """
    if not isinstance(item, pytest.Function):
        yield item
    obj = item.obj
    if is_generator(obj):
        for number, yielded in enumerate(obj()):
            index, call, args = interpret_yielded_test(yielded, number)
            test = pytest.Function(item.name+index, item.parent, args=args, callobj=call)
            yield test
    else:
        yield item


def interpret_yielded_test(obj, number):
    """
    Process an item yielded from a generator.  If the item is named,
    then set the index to "['name']", otherwise set it to "[number]".
    Return the index, the callable and the arguments to the callable.
    """
    if not isinstance(obj, (tuple, list)):
        obj = (obj,)
    if not callable(obj[0]):
        index = "['%s']"%obj[0]
        obj = obj[1:]
    else:
        index = "[%d]"%number
    call, args = obj[0], obj[1:]
    return index, call, args

The above may not work if pytest has changed too much since version 3.2.1. Instead, copy and modify the latest version of _pytest.main.pytest_runtestloop as appropriate; this should provide time for your project to migrate gradually away from yield-based test cases, or at least to yield-based test cases that can be gathered at collection time.