How can I modify a customized build class derived from build_py so that it builds in a temporary directory?

Question

I have a setup.py file with a customized build class derived from build_py, which is imported from either distutils (for older setups) or setuptools:

try:
    warnings.filterwarnings('ignore', 
        message='.*distutils package is deprecated.*', 
        category=DeprecationWarning)
    from distutils.core import setup
    from distutils.command.build_py import build_py
except:
    from setuptools import setup
    from setuptools.command.build_py import build_py

class MyBuild(build_py):

    #... existing custom code here...

    #... new custom code here...

    def run(self):
        build_py.run(self)

if __name__ == '__main__':

    setup(
        name="...",
        version="...",
        #...
        cmdclass={'build_py': MyBuild}
    )

I've run into the problem that when running from a VM client (though oddly not when running outside the client), pip install . fails, apparently because it tries to set up a build directory within the source directory and this fails due to permissions issues. Other people have found pip install . failing under different circumstances. For what it's worth, I'm using Python 3.11 on Windows, and I've seen the failure with both pip 22.3.1 and pip 23.3. (UPDATE: I see the same problems with Python 3.9 and 3.10. Using older versions of pip causes another problem: "ImportError: cannot import name 'Mapping' from 'collections".)

I was thinking that I could retrieve the name of a temporary directory via tempfile.gettempdir() and set the build directory to this value. The problem is that I can't quite figure out how to do this based on the distutils documentation, the Setup tools documentation, and existing examples elsewhere on the web, including Stack Overflow. In particular, I'm not clear on how initialize_options, finalize_options, and set_undefined_options are supposed to work and whether I should be using set_undefined_options at all.

I tried this code, inserted where "#... new custom code here..." is located in my example above:

    def initialize_options(self):
        self.build_base = None
        self.build_lib = None
        print('Set self.build_base and self.build_lib to None')
        
    def finalize_options(self):
        if not self.build_base:
            self.build_base = tempfile.gettempdir()
            print('Set self.build_base to ', self.build_base)
            
        if not self.build_lib:
            self.build_lib = os.path.join(self.build_base, 'lib')
            print('Set self.build_lib to ', self.build_lib)

But then I got an exception when I executed pip install . in the directory containing the setup.py file:

  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      Set self.build_base and self.build_lib to None
      Set self.build_base to  <...>\AppData\Local\Temp
      Set self.build_lib to  <...>\AppData\Local\Temp\lib
      Traceback (most recent call last):
        File "C:\pyvenv\<...>\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 351, in <module>
          main()
        File "C:\pyvenv\<...>\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\pyvenv\<...>\Lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 138, in <module>
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\dist.py", line 989, in run_command
          super().run_command(command)
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 318, in run
          self.find_sources()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 326, in find_sources
          mm.run()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 548, in run
          self.add_defaults()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 586, in add_defaults
          sdist.add_defaults(self)
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\sdist.py", line 113, in add_defaults
          super().add_defaults()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 249, in add_defaults
          self._add_defaults_python()
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\command\sdist.py", line 125, in _add_defaults_python
          self.filelist.extend(build_py.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\command\build_py.py", line 303, in get_source_files
          return [module[-1] for module in self.find_all_modules()]
                                           ^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\command\build_py.py", line 293, in find_all_modules
          if self.py_modules:
             ^^^^^^^^^^^^^^^
        File "C:\<...>\AppData\Local\Temp\pip-build-env-42o2vdhl\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: py_modules. Did you mean: 'find_modules'?
      [end of output]

I tried adding this line to the end of finalize_options:

        # self.set_undefined_options('build_py', ('build_base', 'build_base'),
            # ('build_lib', 'build_lib'))

but this just got me into a nasty recursive loop.

What should I be doing instead?