Sphinx warning: Failed to import test.test_adder from module `pythontemplate`

Question

Context

After creating a root_dir/docs/source/conf.py that automatically generates the .rst files for each .py file in the root_dir/src (and root_dir/test/) directory (and its children), I am experiencing some difficulties linking to the src/projectname/__main__.py and root_dir/test/<test files>.py within the .rst files.

The repository structure follows:

src/projectname/__main__.py
src/projectname/helper.py
test/test_adder.py
docs/source/conf.py

(where projectname is: pythontemplate.)

Error message

When I build the Sphinx documentation using: cd docs && make html, I get the following "warning":

WARNING: Failed to import pythontemplate.test.test_adder.
Possible hints:
* AttributeError: module 'pythontemplate' has no attribute 'test'
* ModuleNotFoundError: No module named 'pythontemplate.test'

...

WARNING: autodoc: failed to import module 'test.test_adder' from module 'pythontemplate'; the following exception was raised:
No module named 'pythontemplate.test'

Design Choices

I know some projects include the test/ files within the src/test and some put the test files into the root dir, the latter is followed in this project. By naming the test directory test instead of tests, they are automatically included in the dist created with pip install -e .. This is verified by opening the:dist/pythontemplate-1.0.tar.gz file and verifying that the pythontemplate-1.0 directory contains the test directory (along with the src directory). However the test directory is not included in the whl file. (This is desired as the users should not have to run the tests, but should be able to do so if they want using the tar.gz).

generated .rst documentation files

For the tests, test/test_adder.py file I generated root_dir/docs/source/autogen/test/test_adder.rst with content:

.. _test_adder-module:

test_adder Module
=================

.. automodule:: test.test_adder
   :members:
   :undoc-members:
   :show-inheritance:

Where it is not able to import the test.test_adder.py file. (I also tried .. automodule:: pythontemplate.test.test_adder though that did not import it either).

Question

How can I refer to the test_<something>.py files in the root_dir/test folder from the (auto-generated) .rst documents in docs/source/autogen/test/test_<something>.rst file, such that Sphinx is able to import it?

Conf.py

For completeness, below is the conf.py file:

"""Configuration file for the Sphinx documentation builder.

For the full list of built-in configuration values, see the documentation:
https://www.sphinx-doc.org/en/master/usage/configuration.html

-- Project information -----------------------------------------------------
https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
"""  #

# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))


# -- Project information -----------------------------------------------------

import os
import shutil
import sys

# This makes the Sphinx documentation tool look at the root of the repository
# for .py files.
from datetime import datetime
from pathlib import Path
from typing import List, Tuple

sys.path.insert(0, os.path.abspath(".."))


def split_filepath_into_three(*, filepath: str) -> Tuple[str, str, str]:
    """Split a file path into directory path, filename, and extension.

    Args:
        filepath (str): The input file path.

    Returns:
        Tuple[str, str, str]: A tuple containing directory path, filename, and
        extension.
    """
    path_obj: Path = Path(filepath)
    directory_path: str = str(path_obj.parent)
    filename = os.path.splitext(path_obj.name)[0]
    extension = path_obj.suffix

    return directory_path, filename, extension


def get_abs_root_path() -> str:
    """Returns the absolute path of the root dir of this repository.

    Throws an error if the current path does not end in /docs/source.
    """
    current_abs_path: str = os.getcwd()
    assert_abs_path_ends_in_docs_source(current_abs_path=current_abs_path)
    abs_root_path: str = current_abs_path[:-11]
    return abs_root_path


def assert_abs_path_ends_in_docs_source(*, current_abs_path: str) -> None:
    """Asserts the current absolute path ends in /docs/source."""
    if current_abs_path[-12:] != "/docs/source":
        print(f"current_abs_path={current_abs_path}")
        raise ValueError(
            "Error, current_abs_path is expected to end in: /docs/source"
        )


def loop_over_files(*, abs_search_path: str, extension: str) -> List[str]:
    """Loop over all files in the specified root directory and its child
    directories.

    Args:
        root_directory (str): The root directory to start the traversal from.
    """
    filepaths: List[str] = []
    for root, _, files in os.walk(abs_search_path):
        for filename in files:
            extension_len: int = -len(extension)
            if filename[extension_len:] == extension:
                filepath = os.path.join(root, filename)
                filepaths.append(filepath)
    return filepaths


def is_unwanted(*, filepath: str) -> bool:
    """Hardcoded filter of unwanted datatypes."""
    base_name = os.path.basename(filepath)
    if base_name == "__init__.py":
        return True
    if base_name.endswith("pyc"):
        return True
    if "something/another" in filepath:
        return True
    return False


def filter_unwanted_files(*, filepaths: List[str]) -> List[str]:
    """Filters out unwanted files from a list of file paths.

    Unwanted files include:
    - Files named "init__.py"
    - Files ending with "swag.py"
    - Files in the subdirectory "something/another"

    Args:
        filepaths (List[str]): List of file paths.

    Returns:
        List[str]: List of filtered file paths.
    """

    return [
        filepath
        for filepath in filepaths
        if not is_unwanted(filepath=filepath)
    ]


def get_abs_python_filepaths(
    *, abs_root_path: str, extension: str, root_folder_name: str
) -> List[str]:
    """Returns all the Python files in this repo."""

    # Get the file lists.
    py_files: List[str] = loop_over_files(
        abs_search_path=f"{abs_root_path}docs/source/../../{root_folder_name}",
        extension=extension,
    )

    # Merge and filter to preserve the relevant files.
    filtered_filepaths: List[str] = filter_unwanted_files(filepaths=py_files)
    return filtered_filepaths


def abs_to_relative_python_paths_from_root(
    *, abs_py_paths: List[str], abs_root_path: str
) -> List[str]:
    """Converts the absolute Python paths to relative Python filepaths as seen
    from the root dir."""
    rel_py_filepaths: List[str] = []

    for abs_py_path in abs_py_paths:

        flattened_filepath = os.path.normpath(abs_py_path)
        print(f"flattened_filepath={flattened_filepath}")
        print(f"abs_root_path={abs_root_path}")
        if abs_root_path not in flattened_filepath:
            print(f"abs_root_path={abs_root_path}")
            print(f"flattened_filepath={flattened_filepath}")
            raise ValueError(
                "Error, root dir should be in flattened_filepath."
            )
        rel_py_filepaths.append(
            os.path.relpath(flattened_filepath, abs_root_path)
        )
    return rel_py_filepaths


def delete_directory(*, directory_path: str) -> None:
    """Deletes a directory and its contents.

    Args:
        directory_path (Union[str, bytes]): Path to the directory to be
        deleted.

    Raises:
        FileNotFoundError: If the specified directory does not exist.
        PermissionError: If the function lacks the necessary permissions to
          delete the directory.
        OSError: If an error occurs while deleting the directory.

    Returns:
        None
    """
    if os.path.exists(directory_path) and os.path.isdir(directory_path):
        shutil.rmtree(directory_path)


def create_relative_path(*, relative_path: str) -> None:
    """Creates a relative path if it does not yet exist.

    Args:
        relative_path (str): Relative path to create.

    Returns:
        None
    """
    if not os.path.exists(relative_path):
        os.makedirs(relative_path)
    if not os.path.exists(relative_path):
        raise NotADirectoryError(f"Error, did not find:{relative_path}")


def create_rst(
    *,
    autogen_dir: str,
    rel_filedir: str,
    filename: str,
    pyproject_name: str,
    py_type: str,
) -> None:
    """Creates a reStructuredText (.rst) file with automodule directives.

    Args:
        rel_filedir (str): Path to the directory where the .rst file will be
        created.
        filename (str): Name of the .rst file (without the .rst extension).

    Returns:
        None
    """
    if py_type == "src":
        prelude: str = pyproject_name
    elif py_type == "test":
        prelude = f"{pyproject_name}.test"
    else:
        raise ValueError(f"Error, py_type={py_type} is not supported.")
    # if filename != "__main__":
    title_underline = "=" * len(f"{filename}-module")

    rst_content = f"""
.. _{filename}-module:

{filename} Module
{title_underline}

.. automodule:: {prelude}.{filename}
   :members:
   :undoc-members:
   :show-inheritance:
"""
    # .. automodule:: {rel_filedir.replace("/", ".")}.{filename}
    rst_filepath: str = os.path.join(
        f"{autogen_dir}{rel_filedir}", f"{filename}.rst"
    )

    with open(rst_filepath, "w", encoding="utf-8") as rst_file:
        rst_file.write(rst_content)


def generate_rst_per_code_file(
    *, extension: str, pyproject_name: str
) -> List[str]:
    """Generates a parameterised .rst file for each .py file of the project, to
    automatically include its documentation in Sphinx.

    Returns rst filepaths.
    """

    abs_root_path: str = get_abs_root_path()
    abs_src_py_paths: List[str] = get_abs_python_filepaths(
        abs_root_path=abs_root_path,
        extension=extension,
        root_folder_name="src",
    )
    abs_test_py_paths: List[str] = get_abs_python_filepaths(
        abs_root_path=abs_root_path,
        extension=extension,
        root_folder_name="test",
    )
    current_abs_path: str = os.getcwd()
    autogen_dir: str = f"{current_abs_path}/autogen/"
    prepare_rst_directories(autogen_dir=autogen_dir)
    rst_paths: List[str] = []

    rst_paths.extend(
        create_rst_files(
            pyproject_name=pyproject_name,
            abs_root_path=abs_root_path,
            autogen_dir=autogen_dir,
            abs_py_paths=abs_src_py_paths,
            py_type="src",
        )
    )
    rst_paths.extend(
        create_rst_files(
            pyproject_name=pyproject_name,
            abs_root_path=abs_root_path,
            autogen_dir=autogen_dir,
            abs_py_paths=abs_test_py_paths,
            py_type="test",
        )
    )
    return rst_paths


def prepare_rst_directories(*, autogen_dir: str) -> None:
    """Creates the output directory for the auto-generated .rst documentation
    files."""
    delete_directory(directory_path=autogen_dir)
    create_relative_path(relative_path=autogen_dir)


def create_rst_files(
    *,
    pyproject_name: str,
    abs_root_path: str,
    autogen_dir: str,
    abs_py_paths: List[str],
    py_type: str,
) -> List[str]:
    """Loops over the python files of py_type src or test, and creates the .rst
    files that point to the actual .py file such that Sphinx can generate its
    documentation on the fly."""
    rel_root_py_paths: List[str] = abs_to_relative_python_paths_from_root(
        abs_py_paths=abs_py_paths, abs_root_path=abs_root_path
    )
    rst_paths: List[str] = []

    # Create file for each py file.
    for rel_root_py_path in rel_root_py_paths:
        rel_filedir: str
        filename: str
        rel_filedir, filename, _ = split_filepath_into_three(
            filepath=rel_root_py_path
        )

        create_relative_path(relative_path=f"{autogen_dir}{rel_filedir}")

        create_rst(
            autogen_dir=autogen_dir,
            rel_filedir=rel_filedir,
            filename=filename,
            pyproject_name=pyproject_name,
            py_type=py_type,
        )

        rst_path: str = os.path.join(f"autogen/{rel_filedir}", f"{filename}")
        rst_paths.append(rst_path)
    return rst_paths


def generate_index_rst(*, filepaths: List[str]) -> str:
    """Generates the list of all the auto-generated rst files."""
    now = datetime.now().strftime("%a %b %d %H:%M:%S %Y")
    content = f"""\
.. jsonmodipy documentation main file, created by
   sphinx-quickstart on {now}.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

.. include:: manual.rst

Auto-generated documentation from Python code
=============================================
.. toctree::
   :maxdepth: 2
"""

    for filepath in filepaths:
        content += f"\n   {filepath}"

    content += """

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
"""

    return content


def write_index_rst(*, filepaths: List[str], output_file: str) -> None:
    """Creates an index.rst file that is used to generate the Sphinx
    documentation."""
    index_rst_content = generate_index_rst(filepaths=filepaths)

    with open(output_file, "w", encoding="utf-8") as index_file:
        index_file.write(index_rst_content)


# Call functions to generate rst Sphinx documentation structure.
# Readthedocs sets it to contents.rst, but it is index.rst in the used example.
# -- General configuration ---------------------------------------------------
project: str = "Decentralised-SAAS-Investment-Structure"
main_doc: str = "index"
PYPROJECT_NAME: str = "pythontemplate"

# pylint:disable=W0622
copyright: str = "2024, a-t-0"
author: str = "a-t-0"

the_rst_paths: List[str] = generate_rst_per_code_file(
    extension=".py", pyproject_name=PYPROJECT_NAME
)
if len(the_rst_paths) == 0:
    raise ValueError(
        "Error, did not find any Python files for which documentation needs"
        + " to be generated."
    )
write_index_rst(filepaths=the_rst_paths, output_file="index.rst")


# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions: List[str] = [
    "sphinx.ext.duration",
    "sphinx.ext.doctest",
    "sphinx.ext.autodoc",
    "sphinx.ext.autosummary",
    "sphinx.ext.intersphinx",
    # Include markdown files in Sphinx documentation
    "myst_parser",
]


# Add any paths that contain templates here, relative to this directory.
templates_path: List[str] = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns: List[str] = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme: str = "alabaster"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path: List[str] = ["_static"]

Note

I am aware the error message is thrown because the tests are not in pythontemplate pip package. As explained above, that is a design choice. The question is about how to import those test files from the .rst file without adding the test into the pip package.

I can import the content of the test_adder.py file in the .rst file that should do the autodoc using:

.. _test_adder-module:

test_adder Module
=================
Hello
=====
.. include:: ../../../../test/test_adder.py
.. automodule:: ../../../../test/test_adder.py
   :members:
   :undoc-members:
   :show-inheritance:

However the automodule does not recognise that path, nor does automodule ........test/test_adder.

Answer 1

Better Answer

Adding the path as suggested in the comments was sufficient. In essence adding this to: conf.py made the test files findable in the Sphinx documentation:

sys.path.insert(0, os.path.abspath("../.."))

Bad Answer

Based on this answer I copied all the .py files from the root_dir/test directory into their identical relative path in root_dir/docs/source/test/ and then compiled html doc and added a command to delete those duplicate files again with:

cd docs && make html && rm -r source/test

That worked with the following conf.py:

"""Configuration file for the Sphinx documentation builder.

For the full list of built-in configuration values, see the documentation:
https://www.sphinx-doc.org/en/master/usage/configuration.html

-- Project information -----------------------------------------------------
https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
"""  #

# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath("../.."))


# -- Project information -----------------------------------------------------

import os
import shutil
import sys

# This makes the Sphinx documentation tool look at the root of the repository
# for .py files.
from datetime import datetime
from pathlib import Path
from typing import List, Tuple


def get_abs_root_path() -> str:
    """Returns the absolute path of the root dir of this repository.

    Throws an error if the current path does not end in /docs/source.
    """
    current_abs_path: str = os.getcwd()
    assert_abs_path_ends_in_docs_source(current_abs_path=current_abs_path)
    abs_root_path: str = current_abs_path[:-11]
    return abs_root_path


def assert_abs_path_ends_in_docs_source(*, current_abs_path: str) -> None:
    """Asserts the current absolute path ends in /docs/source."""
    if current_abs_path[-12:] != "/docs/source":
        print(f"current_abs_path={current_abs_path}")
        raise ValueError(
            "Error, current_abs_path is expected to end in: /docs/source"
        )


# sys.path.insert(0, os.path.abspath(f"{get_abs_root_path()}/test"))


def split_filepath_into_three(*, filepath: str) -> Tuple[str, str, str]:
    """Split a file path into directory path, filename, and extension.

    Args:
        filepath (str): The input file path.

    Returns:
        Tuple[str, str, str]: A tuple containing directory path, filename, and
        extension.
    """
    path_obj: Path = Path(filepath)
    directory_path: str = str(path_obj.parent)
    filename = os.path.splitext(path_obj.name)[0]
    extension = path_obj.suffix

    return directory_path, filename, extension


def loop_over_files(*, abs_search_path: str, extension: str) -> List[str]:
    """Loop over all files in the specified root directory and its child
    directories.

    Args:
        root_directory (str): The root directory to start the traversal from.
    """
    filepaths: List[str] = []
    for root, _, files in os.walk(abs_search_path):
        for filename in files:
            extension_len: int = -len(extension)
            if filename[extension_len:] == extension:
                filepath = os.path.join(root, filename)
                filepaths.append(filepath)
    return filepaths


def is_unwanted(*, filepath: str) -> bool:
    """Hardcoded filter of unwanted datatypes."""
    base_name = os.path.basename(filepath)
    if base_name == "__init__.py":
        return True
    if base_name.endswith("pyc"):
        return True
    if "something/another" in filepath:
        return True
    return False


def filter_unwanted_files(*, filepaths: List[str]) -> List[str]:
    """Filters out unwanted files from a list of file paths.

    Unwanted files include:
    - Files named "init__.py"
    - Files ending with "swag.py"
    - Files in the subdirectory "something/another"

    Args:
        filepaths (List[str]): List of file paths.

    Returns:
        List[str]: List of filtered file paths.
    """

    return [
        filepath
        for filepath in filepaths
        if not is_unwanted(filepath=filepath)
    ]


def get_abs_python_filepaths(
    *, abs_root_path: str, extension: str, root_folder_name: str
) -> List[str]:
    """Returns all the Python files in this repo."""

    # Get the file lists.
    py_files: List[str] = loop_over_files(
        abs_search_path=f"{abs_root_path}docs/source/../../{root_folder_name}",
        extension=extension,
    )

    # Merge and filter to preserve the relevant files.
    filtered_filepaths: List[str] = filter_unwanted_files(filepaths=py_files)
    return filtered_filepaths


def abs_to_relative_python_paths_from_root(
    *, abs_py_paths: List[str], abs_root_path: str
) -> List[str]:
    """Converts the absolute Python paths to relative Python filepaths as seen
    from the root dir."""
    rel_py_filepaths: List[str] = []

    for abs_py_path in abs_py_paths:

        flattened_filepath = os.path.normpath(abs_py_path)

        if abs_root_path not in flattened_filepath:
            print(f"abs_root_path={abs_root_path}")
            print(f"flattened_filepath={flattened_filepath}")
            raise ValueError(
                "Error, root dir should be in flattened_filepath."
            )
        rel_py_filepaths.append(
            os.path.relpath(flattened_filepath, abs_root_path)
        )
    return rel_py_filepaths


def delete_directory(*, directory_path: str) -> None:
    """Deletes a directory and its contents.

    Args:
        directory_path (Union[str, bytes]): Path to the directory to be
        deleted.

    Raises:
        FileNotFoundError: If the specified directory does not exist.
        PermissionError: If the function lacks the necessary permissions to
          delete the directory.
        OSError: If an error occurs while deleting the directory.

    Returns:
        None
    """
    if os.path.exists(directory_path) and os.path.isdir(directory_path):
        shutil.rmtree(directory_path)


def create_relative_path(*, relative_path: str) -> None:
    """Creates a relative path if it does not yet exist.

    Args:
        relative_path (str): Relative path to create.

    Returns:
        None
    """
    if not os.path.exists(relative_path):
        os.makedirs(relative_path)
    if not os.path.exists(relative_path):
        raise NotADirectoryError(f"Error, did not find:{relative_path}")


def create_rst(
    *,
    autogen_dir: str,
    rel_filedir: str,
    filename: str,
    pyproject_name: str,
    py_type: str,
) -> None:
    """Creates a reStructuredText (.rst) file with automodule directives.

    Args:
        rel_filedir (str): Path to the directory where the .rst file will be
        created.
        filename (str): Name of the .rst file (without the .rst extension).

    Returns:
        None
    """
    if py_type == "src":
        # prelude: str = f"{pyproject_name}."

        if rel_filedir[:4] != "src/":
            raise ValueError(
                "Expected relative file dir for src files to start with:src/"
            )
        prelude = f"{rel_filedir[4:]}.".replace("/", ".")
    elif py_type == "test":
        prelude = f"{rel_filedir}.".replace("/", ".")
    else:
        raise ValueError(f"Error, py_type={py_type} is not supported.")
    # if filename != "__main__":
    title_underline = "=" * len(f"{filename}-module")

    rst_content = f"""
.. _{filename}-module:

{filename} Module
{title_underline}

.. automodule:: {prelude}{filename}
   :members:
   :undoc-members:
   :show-inheritance:
"""
    rst_filepath: str = os.path.join(
        f"{autogen_dir}{rel_filedir}", f"{filename}.rst"
    )

    with open(rst_filepath, "w", encoding="utf-8") as rst_file:
        rst_file.write(rst_content)


def generate_rst_per_code_file(
    *, extension: str, pyproject_name: str
) -> List[str]:
    """Generates a parameterised .rst file for each .py file of the project, to
    automatically include its documentation in Sphinx.

    Returns rst filepaths.
    """

    abs_root_path: str = get_abs_root_path()
    abs_src_py_paths: List[str] = get_abs_python_filepaths(
        abs_root_path=abs_root_path,
        extension=extension,
        root_folder_name="src",
    )
    abs_test_py_paths: List[str] = get_abs_python_filepaths(
        abs_root_path=abs_root_path,
        extension=extension,
        root_folder_name="test",
    )
    current_abs_path: str = os.getcwd()
    autogen_dir: str = f"{current_abs_path}/autogen/"
    prepare_rst_directories(autogen_dir=autogen_dir)
    rst_paths: List[str] = []

    rst_paths.extend(
        create_rst_files(
            pyproject_name=pyproject_name,
            abs_root_path=abs_root_path,
            autogen_dir=autogen_dir,
            abs_py_paths=abs_src_py_paths,
            py_type="src",
        )
    )
    rst_paths.extend(
        create_rst_files(
            pyproject_name=pyproject_name,
            abs_root_path=abs_root_path,
            autogen_dir=autogen_dir,
            abs_py_paths=abs_test_py_paths,
            py_type="test",
        )
    )
    return rst_paths


def prepare_rst_directories(*, autogen_dir: str) -> None:
    """Creates the output directory for the auto-generated .rst documentation
    files."""
    delete_directory(directory_path=autogen_dir)
    create_relative_path(relative_path=autogen_dir)


def create_rst_files(
    *,
    pyproject_name: str,
    abs_root_path: str,
    autogen_dir: str,
    abs_py_paths: List[str],
    py_type: str,
) -> List[str]:
    """Loops over the python files of py_type src or test, and creates the .rst
    files that point to the actual .py file such that Sphinx can generate its
    documentation on the fly."""
    rel_root_py_paths: List[str] = abs_to_relative_python_paths_from_root(
        abs_py_paths=abs_py_paths, abs_root_path=abs_root_path
    )
    rst_paths: List[str] = []

    # Create file for each py file.
    for rel_root_py_path in rel_root_py_paths:
        if "__main__.py" not in rel_root_py_path:
            rel_filedir: str
            filename: str
            rel_filedir, filename, _ = split_filepath_into_three(
                filepath=rel_root_py_path
            )

            create_relative_path(relative_path=f"{autogen_dir}{rel_filedir}")

            create_rst(
                autogen_dir=autogen_dir,
                rel_filedir=rel_filedir,
                filename=filename,
                pyproject_name=pyproject_name,
                py_type=py_type,
            )

            rst_path: str = os.path.join(
                f"autogen/{rel_filedir}", f"{filename}"
            )
            rst_paths.append(rst_path)
    return rst_paths


def generate_index_rst(*, filepaths: List[str]) -> str:
    """Generates the list of all the auto-generated rst files."""
    now = datetime.now().strftime("%a %b %d %H:%M:%S %Y")

    # TODO: make filepaths relative nested directories.
    content = f"""\
.. jsonmodipy documentation main file, created by
   sphinx-quickstart on {now}.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

.. include:: manual.rst

Auto-generated documentation from Python code
=============================================
.. toctree::
   :maxdepth: 2
"""

    for filepath in filepaths:
        content += f"\n   {filepath}"

    content += """

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
"""

    return content


def write_index_rst(*, filepaths: List[str], output_file: str) -> None:
    """Creates an index.rst file that is used to generate the Sphinx
    documentation."""
    index_rst_content = generate_index_rst(filepaths=filepaths)

    with open(output_file, "w", encoding="utf-8") as index_file:
        index_file.write(index_rst_content)


def replicate_dir_structure(
    *, abs_path: str, abs_target_dir: str
) -> List[str]:
    """Replicates the directory structure from abs_path to target_dir,
    including nested folders.

    Args:
        abs_path: The absolute path of the source directory.
        target_dir: The absolute path of the destination directory where the structure will be recreated.
    """
    relative_target_subpaths: List[str] = []
    for root, dirs, files in os.walk(abs_path):

        rel_path = os.path.relpath(
            root, abs_path
        )  # Get relative path from abs_path
        target_subpath = os.path.join(
            abs_target_dir, rel_path
        )  # Create corresponding subpath in target_dir
        if rel_path not in [".", "__pycache__"]:

            # if "__pycache__" not in target_subpath:

            os.makedirs(
                target_subpath, exist_ok=True
            )  # Create nested folders if needed
            # target_subpaths.append(target_subpath)

            relative_target_subpaths.append(f"/{rel_path}")
    return relative_target_subpaths


def list_directory_files(*, abs_path: str) -> list[str]:
    """Lists all files directly in the specified absolute path.

    Args:
        abs_path: The absolute path of the directory to list files from.

    Returns:
        A list of filenames (without full paths) directly in the directory.
    """
    entries = os.listdir(abs_path)
    return [f for f in entries if os.path.isfile(os.path.join(abs_path, f))]


def copy_py_files_in_dir(
    *, abs_testfile_path: str, abs_dest_path: str, target_subpath: str
):

    for file in list_directory_files(abs_path=abs_testfile_path):
        if file.endswith(".py"):
            src_file = os.path.join(abs_testfile_path, file)
            dst_file = os.path.join(f"{abs_dest_path}{target_subpath}", file)
            # os.copy(src_file, dst_file)  # Use replace to ensure overwrite
            shutil.copy2(src_file, dst_file)


def copy_py_files(
    *, abs_root_path: str, original_test_dir: str, dest_dir: str
) -> None:
    """Copies all .py files from original_test_dir to dst_dir, including nested
    folders.

    Args:
        original_test_dir: The test directory containing the .py files.
        dst_dir: The destination directory where the files will be copied.
    """
    abs_dest_path: str = f"{abs_root_path}{dest_dir}"
    abs_test_path: str = f"{abs_root_path}{original_test_dir}"
    delete_directory(directory_path=abs_dest_path)
    create_relative_path(relative_path=abs_dest_path)
    target_subpaths: List[str] = replicate_dir_structure(
        abs_path=abs_test_path, abs_target_dir=abs_dest_path
    )

    for target_subpath in target_subpaths:
        abs_testfile_path: str = f"{abs_test_path}{target_subpath}"
        copy_py_files_in_dir(
            abs_testfile_path=abs_testfile_path,
            abs_dest_path=abs_dest_path,
            target_subpath=target_subpath,
        )
    copy_py_files_in_dir(
        abs_testfile_path=abs_test_path,
        abs_dest_path=abs_dest_path,
        target_subpath="",
    )


# Call functions to generate rst Sphinx documentation structure.
# Readthedocs sets it to contents.rst, but it is index.rst in the used example.
# -- General configuration ---------------------------------------------------
project: str = "Decentralised-SAAS-Investment-Structure"
main_doc: str = "index"
PYPROJECT_NAME: str = "pythontemplate"

# pylint:disable=W0622
copyright: str = "2024, a-t-0"
author: str = "a-t-0"

the_rst_paths: List[str] = generate_rst_per_code_file(
    extension=".py", pyproject_name=PYPROJECT_NAME
)
if len(the_rst_paths) == 0:
    raise ValueError(
        "Error, did not find any Python files for which documentation needs"
        + " to be generated."
    )
write_index_rst(filepaths=the_rst_paths, output_file="index.rst")


# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions: List[str] = [
    "sphinx.ext.duration",
    "sphinx.ext.doctest",
    "sphinx.ext.autodoc",
    "sphinx.ext.autosummary",
    "sphinx.ext.intersphinx",
    # Include markdown files in Sphinx documentation
    "myst_parser",
]


# Add any paths that contain templates here, relative to this directory.
templates_path: List[str] = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns: List[str] = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme: str = "alabaster"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path: List[str] = ["_static"]

# Horrible hack: copy all .py files of /test/ dir into /docs/source/test.
abs_root_path: str = get_abs_root_path()
copy_py_files(
    abs_root_path=abs_root_path,
    original_test_dir="test",
    dest_dir="docs/source/test",
)
# delete_directory(directory_path=f"{abs_root_path}docs/source/test")

I was not able to use the delete_directory( function in the conf.py because the make html command was executed after the conf.py which meant the duplicate test files would hvae been created and deleted again before make html was able to find them. Before I realised I could delete the duplicate test files, I also had to modify the linters and pytest in pyproject.toml to ignore the duplicate files with:

[tool.mypy]
ignore_missing_imports = true
exclude = ["^docs/source/test/"]

[tool.pytest.ini_options]
# Runs coverage.py through use of the pytest-cov plugin
# An xml report is generated and results are output to the terminal

# TODO: Disable this line to disable CLI coverage reports when running tests.
# addopts = "--cov --cov-report xml:cov.xml --cov-report term"

# Sets the minimum allowed pytest version
minversion = 5.0
# Sets the path where test files are located (Speeds up Test Discovery)
testpaths = ["test"]
pytest_ignore = ["docs/source/test"]

Then I found out I could add the rm -r source/test delete instruction after the make html instruction. I am happy someone suggested the oneliner above to resolve this issue.