pytest order test execution by grouping tests based on markers

Question

I am looking to execute tests cases in pytest using multiple markers (say m1, m2), such that all test cases marked with m1 will run first, and then test cases marked m2 will run after. I am using the command: pytest -s -m "m1 or m2" to run pytest here. Example: test_a uses marker m1, test_b uses marker m2, test_c uses marker m1, test_d uses marker m2.

If I execute pytest -s -m "m1 or m2", the test execution order is: test_a->test_b->test_c->test_d because pytest executes them in an alphabetical order. But I want the execution to be test_a->test_c->test_b->test_d such that all tests marked m1 are executed first, and then the tests marked m2 are executed after.

import pytest

@pytest.mark.m1
def test_a():
    pass

@pytest.mark.m2
def test_b():
    pass

@pytest.mark.m1
def test_c():
    pass

@pytest.mark.m2
def test_d():
    pass

Answer 1

pytest -m "custom_marker" runs all tests with that marker it does not change execution order.

I recommend looking into the pytest-order plugin. This allows the use of a @pytest.marker.order(#) marker and can even be applied to the class level. So you could have all your m1 tests in one TestClass and all the m2 tests in another.

import pytest

@pytest.marker.order(1)
class TestM1:

    def test_a(self):
        pass

    def test_c(self):
        pass

@pytest.marker.order(2)
class TestM2:

    def test_b(self):
        pass

    def test_d(self):
        pass

Alternatively you can try looking into the "pytest_collection_modifyitems" hook, but I am not very familiar with using it.

Answer 2

Getting a clue from @cdub's answer regarding pytest_collection_modifyitems, I could implement the following and got it to work. Reference to original implementaion from: https://stackoverflow.com/questions/70738211/run-pytest-classes-in-custom-order#:~:text=You%20can%20use%20the%20pytest_collection_modifyitems,allows%20to%20sort%20by%20class.

def pytest_collection_modifyitems(items):
    """Modifies test items in place to ensure test functions run in a given order"""
    marker_order = ["m1", "m2"]
    marker_mapping = {}
    for item in items:
        if val := [x.name for x in item.own_markers if x.name in marker_order]:
            marker_mapping[item] = val[0]

    sorted_items = items.copy()
    for mark_ in marker_order:
        sorted_items = [it for it in sorted_items if it in marker_mapping and marker_mapping[it] != mark_] + \
                       [it for it in sorted_items if it in marker_mapping and marker_mapping[it] == mark_]

    items[:] = sorted_items

Answer 3

For completeness, there is now also an option to achieve this with pytest-order. The option --order-marker-prefix now allows you to sort by your custom marker, while you are still able to filter by marker.

For your example, you can run the tests using

pytest --order-marker-prefix=m -s -m "m1 or m2"

to run the tests marked with "m1" before the tests marked with "m2". In this case the tests are handled the same way as if they had an additional order marker with the number as index, e.g. @pytest.mark.m2 is handled like @pytest.marker.order(2).

Disclaimer: I'm the maintainer of pytest-order. This question inspired me to add this feature.