How to generate a github actions build matrix that dynamically includes a list of nox sessions?

Question

I recently started to migrate some of my open source python libraries from Travis to Github Actions. To be more independent from the CI/CD platform, I decided to first describe all test environments and configurations using nox.

Let's consider the following prototypical noxfile.py :

import nox

@nox.session(python=["3.7", "3.8"])
def tests(session):
    print("Interesting stuff going on here")

@nox.session(python=["3.7", "3.8"])
@nox.parametrize("a", [-1, 1])
@nox.parametrize("b", [False, True])
def tests_with_params(session, a, b):
    print("Interesting stuff going on here")

@nox.session(python="3.7")
def other(session):
    print("another session")

Leading to the following nox --list output:

* tests-3.7
* tests-3.8
* tests_with_params-3.7(b=False, a=-1)
* tests_with_params-3.7(b=True, a=-1)
* tests_with_params-3.7(b=False, a=1)
* tests_with_params-3.7(b=True, a=1)
* tests_with_params-3.8(b=False, a=-1)
* tests_with_params-3.8(b=True, a=-1)
* tests_with_params-3.8(b=False, a=1)
* tests_with_params-3.8(b=True, a=1)
* other

I would like the github actions workflow to run all sessions for tests, or for tests_with_params. Using a hardcoded build matrix in my workflow yaml definition file it works, for example here I list the two sessions for tests:

# (in .github/workflows/base.yml)
jobs:
  run_all_tests:
    strategy:
      fail-fast: false
      matrix:
        # Here we manually list two nox sessions
        nox_session: ["tests-3.7", "tests-3.8"]
    name: Run nox session ${{ matrix.nox_session }}
    runs-on: ubuntu-latest
    steps:
      # (other steps before this: checkout code, install python and nox...)
      - run: nox -s "${{ matrix.nox_session }}"

however as can be seen above, the list of sessions has to be copied manually in the matrix. So if I decide to change the parametrization of my nox sessions, this will have to be updated.

It would therefore be far easier to have the github actions workflow "ask nox" to dynamically get this list. How can we do this ?

Answer 1

(Answering my own question since I could not find an entry on SO)

I found a way to do this thanks to this great post. The solution is based on two steps:

1. first create a nox task gha_list that will print the list of all session names for a given base session name
2. then add a job to the github action workflow that will leverage this task to get the list dynamically, and inject this list into the subsequent job's build matrix.

1. Create a nox task to print the desired list of sessions

Let's add this task to our noxfile.py:

import itertools
import json

@nox.session(python=False)
def gha_list(session):
    """(mandatory arg: <base_session_name>) Prints all sessions available for <base_session_name>, for GithubActions."""

    # get the desired base session to generate the list for
    if len(session.posargs) != 1:
        raise ValueError("This session has a mandatory argument: <base_session_name>")
    session_func = globals()[session.posargs[0]]

    # list all sessions for this base session
    try:
        session_func.parametrize
    except AttributeError:
        sessions_list = ["%s-%s" % (session_func.__name__, py) for py in session_func.python]
    else:
        sessions_list = ["%s-%s(%s)" % (session_func.__name__, py, param)
                         for py, param in itertools.product(session_func.python, session_func.parametrize)]

    # print the list so that it can be caught by GHA.
    # Note that json.dumps is optional since this is a list of string.
    # However it is to remind us that GHA expects a well-formatted json list of strings.
    print(json.dumps(sessions_list))

We can test that it works in the terminal:

>>> nox -s gha_list -- tests

nox > Running session gha_list
["tests-3.7", "tests-3.8"]
nox > Session gha_list was successful.

>>> nox -s gha_list -- tests_with_params

nox > Running session gha_list
["tests_with_params-3.7(b=False, a=-1)", "tests_with_params-3.7(b=True, a=-1)", "tests_with_params-3.7(b=False, a=1)", "tests_with_params-3.7(b=True, a=1)", "tests_with_params-3.8(b=False, a=-1)", "tests_with_para
ms-3.8(b=True, a=-1)", "tests_with_params-3.8(b=False, a=1)", "tests_with_params-3.8(b=True, a=1)"]
nox > Session gha_list was successful.

2. Edit the github actions workflow to inject this list in the build matrix dynamically

Now that we are able to print the desired list, we modify the github actions workflow to add a job calling it before the job running the tests.

# (in .github/workflows/base.yml)
jobs:
  list_nox_test_sessions:
    runs-on: ubuntu-latest
    steps:
        # (other steps before this: checkout code, install python and nox...)

        - name: list all test sessions
          id: set-matrix
          run: echo "::set-output name=matrix::$(nox -s gha_list -- tests)"

    # save the list into the outputs
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}

  run_all_tests:
    needs: list_nox_test_sessions
    strategy:
      fail-fast: false
      matrix:
        nox_session: ${{ fromJson(needs.list_nox_test_sessions.outputs.matrix) }}
    name: Run nox session ${{ matrix.nox_session }}
    runs-on: ubuntu-latest
    steps:
      # (other steps before this: checkout code, install python)
      - run: nox -s "${{ matrix.nox_session }}"

Example

An example can be found at this repo, with the following workflow:

Other viable alternatives

I also considered using grep on the nox --list output, this would probably work as well. However it might be harder to debug for developers who are at ease with python but not at ease with grep.