"c1xx: fatal error C1083: Cannot open source file: 'cld3/pycld3.cpp': No such file or directory" when installing pycld3 with pip

Question

I am trying to install cld3 with pip on Python 3.9. I have successfully installed protoc and also got the C++ build tools with Visual Studio, however when I run py -m pip install -U pycld3 it fails with a few messages. I am on windows 10 running Python 3.9 It initially fails to build using wheel:

Collecting pycld3
  Using cached pycld3-0.21.tar.gz (652 kB)
Building wheels for collected packages: pycld3
  Building wheel for pycld3 (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\Charlie\AppData\Local\Programs\Python\Python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"'; __file__='"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\Charlie\AppData\Local\Temp\pip-wheel-_9ez3o7a'
       cwd: C:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\
  Complete output (23 lines):
  running bdist_wheel
  running build
  Creating dirs at C:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/cld_3/protos/

This is not surprising as it mentions in the docs that wheel installation is not supported for my OS. However I really do not understand what happens next:

 Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=C:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/cld_3/protos/
  running build_py
  creating build
  creating build\lib.win-amd64-3.9
  creating build\lib.win-amd64-3.9\cld3
  copying cld3\__init__.py -> build\lib.win-amd64-3.9\cld3
  running build_ext
  building 'cld3._cld3' extension
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  creating build\temp.win-amd64-3.9\Release\cld3
  creating build\temp.win-amd64-3.9\Release\src
  creating build\temp.win-amd64-3.9\Release\src\cld_3
  creating build\temp.win-amd64-3.9\Release\src\cld_3\protos
  creating build\temp.win-amd64-3.9\Release\src\script_span
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/usr/local/include/ -IC:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/ -IC:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/cld_3/protos/ -IC:\Users\Charlie\AppData\Local\Programs\Python\Python39\include -IC:\Users\Charlie\AppData\Local\Programs\Python\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /Tpcld3/pycld3.cpp /Fobuild\temp.win-amd64-3.9\Release\cld3/pycld3.obj -std=c++11
  cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
  pycld3.cpp
  c1xx: fatal error C1083: Cannot open source file: 'cld3/pycld3.cpp': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for pycld3
  Running setup.py clean for pycld3
Failed to build pycld3
Installing collected packages: pycld3
    Running setup.py install for pycld3 ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\Charlie\AppData\Local\Programs\Python\Python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"'; __file__='"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Charlie\AppData\Local\Temp\pip-record-fkxpsotm\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\Charlie\AppData\Local\Programs\Python\Python39\Include\pycld3'
         cwd: C:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\
    Complete output (22 lines):
    running install
    running build
    Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=C:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/cld_3/protos/
    running build_py
    creating build
    creating build\lib.win-amd64-3.9
    creating build\lib.win-amd64-3.9\cld3
    copying cld3\__init__.py -> build\lib.win-amd64-3.9\cld3
    running build_ext
    building 'cld3._cld3' extension
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\cld3
    creating build\temp.win-amd64-3.9\Release\src
    creating build\temp.win-amd64-3.9\Release\src\cld_3
    creating build\temp.win-amd64-3.9\Release\src\cld_3\protos
    creating build\temp.win-amd64-3.9\Release\src\script_span
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/usr/local/include/ -IC:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/ -IC:\Users\Charlie\AppData\Local\Temp\pip-install-hl9cd_fj\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\src/cld_3/protos/ -IC:\Users\Charlie\AppData\Local\Programs\Python\Python39\include -IC:\Users\Charlie\AppData\Local\Programs\Python\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /Tpcld3/pycld3.cpp /Fobuild\temp.win-amd64-3.9\Release\cld3/pycld3.obj -std=c++11
    cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
    pycld3.cpp
    c1xx: fatal error C1083: Cannot open source file: 'cld3/pycld3.cpp': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\Charlie\AppData\Local\Programs\Python\Python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"'; __file__='"'"'C:\\Users\\Charlie\\AppData\\Local\\Temp\\pip-install-hl9cd_fj\\pycld3_6beaeb0f5dd847eab99d4f8bff3bd60a\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Charlie\AppData\Local\Temp\pip-record-fkxpsotm\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\Charlie\AppData\Local\Programs\Python\Python39\Include\pycld3' Check the logs for full command output.

As far as I can tell it is because it cannot find "cld3/pycld3.cpp"". Do I need to run an earlier version of Python? Thanks in advance

Answer 1

This is a bug in pycld3: they've forgotten to include the source into sdist. Please report the bug.

Meanwhile the workaround is to install Cython:

pip install Cython
pip install pycld3

Answer 2

[pycld3 maintainer] This issue should be resolved in pycld3 release 0.22.

python3 -m pip install -U pycld3

As @phd points out, this was an error with the distribution upload in 0.21 that neglected to include the intermediate .cpp file. Inclusion of that file in the sdist should not be dependent on the build-time logic of whether a user has or wants to use Cython.

Note that in the 0.22 release, the installation will respect the environment variable USE_CYTHON=1. You must have an existing install of Cython to use this flag. It will force re-compilation from the source .pyx file. This follows the behavior recommended at Distributing Cython Modules.

0.22 also drops support for end-of-life Python 3.5 and no longer passes the unsupported -std=c++11 on Windows MSVC.

Detail: full changelog from 0.21 to 0.22.