VSCode Python unittest discovery fails: 'utils' is not a package -> sys.path ordering?

Question

since a few days, Visual Studio Code Python unittest discovery no longer works for me in several projects that contain a package called utils:

2023-10-18 09:37:36.901 [info] Discovering unittest tests with arguments: /home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/unittestadapter/discovery.py,--udiscovery,-v,-s,./tests,-p,test_*.py
...
ModuleNotFoundError: No module named 'utils.export_to_dict_helper'; 'utils' is not a package

It works as soon as I rename the whole utils folder to something else, like helper_utils. Running the tests from the terminal using

python3 -m unittest discover -v tests

works just fine, even with the directory still named utils. I already tried to downgrade the "Python" extension by a few months and switching to another Python version (tried 3.8 and 3.11 so far), but the error is consistent.

The top of the test file looks like this (print of sys.path for debugging):

import sys
print(sys.path)

# Other (working) imports...

from utils.data_storage_selector import DataStorageSelector
from utils.export_to_dict_helper import ExportToDictHelper

The full output of the test discovery looks like this:

2023-10-18 09:37:36.901 [info] Discovering unittest tests with arguments: /home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/unittestadapter/discovery.py,--udiscovery,-v,-s,./tests,-p,test_*.py

2023-10-18 09:37:36.901 [info] > ~/.pyenv/versions/3.8.16/bin/python ~/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/unittestadapter/discovery.py --udiscovery -v -s ./tests -p test_*.py
2023-10-18 09:37:36.901 [info] cwd: .
2023-10-18 09:37:37.218 [info] [
    '/home/phip/.../poleno-event-model/tests',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/lib/python',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/unittestadapter',
    '/home/phip/.../poleno-event-model',
    '/home/phip/.pyenv/versions/3.8.16/lib/python38.zip',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/site-packages',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/site-packages/CharPyLS-1.0.3-py3.8-linux-x86_64.egg',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/lib/python',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles',
    '/home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/lib/python'
]

2023-10-18 09:37:37.498 [info] Test server connected to a client.
2023-10-18 09:37:37.571 [error] Unittest test discovery error 
 Failed to import test module: test_data_model_base
Traceback (most recent call last):
  File "/home/phip/.pyenv/versions/3.8.16/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/phip/.pyenv/versions/3.8.16/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/phip/.../poleno-event-model/tests/test_data_model_base.py", line 20, in <module>
    from poleno_event_model.data_model_base import DataModelBase
  File "/home/phip/.../poleno-event-model/poleno_event_model/__init__.py", line 18, in <module>
    from .event_model_decl import event_model
  File "/home/phip/.../poleno-event-model/poleno_event_model/event_model_decl.py", line 44, in <module>
    from utils.export_to_dict_helper import AttributeFilter
ModuleNotFoundError: No module named 'utils.export_to_dict_helper'; 'utils' is not a package


Failed to import test module: test_poleno_event_model
Traceback (most recent call last):
  File "/home/phip/.pyenv/versions/3.8.16/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/phip/.pyenv/versions/3.8.16/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/phip/.../poleno-event-model/tests/test_poleno_event_model.py", line 26, in <module>
    from poleno_event_model import BackgroundData, MeasurementEvent
  File "/home/phip/.../poleno-event-model/poleno_event_model/__init__.py", line 18, in <module>
    from .event_model_decl import event_model
  File "/home/phip/.../poleno-event-model/poleno_event_model/event_model_decl.py", line 44, in <module>
    from utils.export_to_dict_helper import AttributeFilter
ModuleNotFoundError: No module named 'utils.export_to_dict_helper'; 'utils' is not a package

For comparison, this is the output when running the tests manually:

python3 -m unittest discover tests
[
    '/home/phip/.../poleno-event-model/tests',
    '/home/phip/.../poleno-event-model',
    '/home/phip/.../poleno-event-model',
    '/home/phip/.pyenv/versions/3.8.16/lib/python38.zip',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/site-packages',
    '/home/phip/.pyenv/versions/3.8.16/lib/python3.8/site-packages/CharPyLS-1.0.3-py3.8-linux-x86_64.egg'
]
................
----------------------------------------------------------------------
Ran 17 tests in 0.192s

OK

The utils directory is located at the root of the project (same level as tests and most other local package directories) and contains an __init__.py file.

In case it matters, tool versions are as follows:

• OS: Ubuntu 23.04
• Python: Tested 3.8 and 3.11 via pyenv
• VSCode: 1.83.1 (same problem with 1.83.0 and no older snap available to compare)
• Python extension: v2023.19.12901009 (pre-release for testing, but same problem with current and past release version)

Progress

While writing the question, I probably discovered the problem. The VSCode extension folder unittestadapter that is put into sys.path during discovery and before the workspace folder looks like this:

ls -alh /home/phip/.vscode/extensions/ms-python.python-2023.19.12901009/pythonFiles/unittestadapter
total 44K
drwxrwxr-x 3 phip phip 4.0K Okt 17 16:18 .
drwxrwxr-x 8 phip phip 4.0K Okt 17 16:17 ..
-rw-rw-r-- 1 phip phip 4.6K Okt 17 16:17 discovery.py
-rw-rw-r-- 1 phip phip  11K Okt 17 16:17 execution.py
-rw-rw-r-- 1 phip phip   94 Okt 17 16:17 __init__.py
drwxrwxr-x 2 phip phip 4.0K Okt 17 16:25 __pycache__
-rw-rw-r-- 1 phip phip 7.9K Okt 17 16:17 utils.py

This utils.py module there shadows my project-local utils package.

Question

With the finding above, the problem boils down to How to fix the ordering of the sys.path elements when running the VSCode unittest discovery so that the workspace folder comes before any extension directories?

The secondary question would be why this suddenly changes and apparently only affects me; at least I wasn't able to find something similar in online searches. If this was a problem with a recent VSCode version, I'd assume the web to be riddled with reports about it.

Update

I still haven't found a solution for the problem, but investigated some more: Apparently, the unittestadapter directory in sys.path should not be there at all, as its internal utils module is used in a qualified way (unittestadapter.utils). So only the pythonFiles path should be added, which is in the list a bit further down.

Interestingly, I can't reproduce the problem when creating a new minimal project (just a main module, a utils package with a single module and a test file); no additional unittestadapter gets added in this case. So it is probably not a problem with the VSCode installation. However, the issue persists with at least two projects containing a utils module. And it does not get fixed by removing the __pycache__ directories or renaming the project folder (to invalidate any cache in VSCode installation/config files).

Answer 1

Although I still haven't found a solution for the problem, there is at least a workaround: Switching in VSCode to using pytest instead of unittest allows the tests to be discovered and run just fine.

All I had to do for this was to remove the unittest-related settings from .vscode/settings.json

    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test*.py"
    ],
    "python.testing.unittestEnabled": true,

and then reloading the window. This caused VSCode to offer the testing configuration, for which I selected pytest this time. It then added those settings:

    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true

No changes to the code were required, since pytest is almost fully compatible with the unittest framework, including running asyncio-Tests with the IsolatedAsyncioTestCase.

I'd still love to see a solution (or at least an explanation) of the root problem with pure unittest discovery, though.

Answer 2

The issue, as other answers have pointed out, is that the VSCode test framework is first in the path and has a file called utils.py. This confounds the path definition, since it tries to treat this file as a module and supersedes your utils module definition. On Windows 11:

In order to get around this, I renamed my local utils module to utils_:

import unittest
from utils_.prompts import GeneratePrompt, PromptConstants


class TestGeneratePrompt(unittest.TestCase):
    def test_simple_generate_prompt(self):
.
. (etc.)
.

This was an "ok" solution for me, since I own the repo and it is quite small. A better fix would be to rename the VSCode utils.py mode to something more unique (probably as a Pull-request to the VSCode project)