How can I install GRPCIO on an Apple M1 Silicon laptop?

Question

Every time I try to install a pip package with GRPCIO as a dependency inside a .direnv project I get a build failure due to my architecture being arm64. How can I work around this while waiting for the GRPCIO crew to release an update?

  Using cached grpcio-1.34.0.tar.gz (21.0 MB)
    ERROR: Command errored out with exit status 1:
     command: /Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"'; __file__='"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-pip-egg-info-z41yqiw6
         cwd: /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/
    Complete output (10 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py", line 359, in <module>
        if mac_target and (pkg_resources.parse_version(mac_target) <
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 113, in parse_version
        return packaging.version.Version(v)
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
        match = self._regex.search(version)
    TypeError: expected string or bytes-like object
    ASM Builds for BoringSSL currently not supported on: macosx-11-arm64
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Answer 1

You no longer need to do any of this. Things just work out the box now that the community support for Apple Silicon has caught up. Happy coding!

Answer 2

Edit: Google now provides official grpcio builds for Apple Silicon Macs, since version 1.51.3. Downloading an unofficial binary is no longer necessary.

Unfortunately, Google does not provide grpcio wheels built for Apple Silicon Macs. As you see from this issue, users have come up with a variety of compiler flags to compile the library on M1, which don't always work.

Tired of these problems, I published the grpcio-mac-arm-build repository, which contains a daily GitHub Actions job that compiles the latest stable grpcio and grpcio-tools releases for Apple Silicon. The compiled wheels for Python 3.8, 3.9, 3.10, and 3.11 are available for download in the "Releases" page. Everything is open sourced and released with the same license of the grpcio library. This way, you don't need to compile the library at all.

Answer 3

The following settings worked for me,

Used https://asdf-vm.com/ to install and manage my python

Used https://direnv.net and created $PWD/.envrc with just one line

layout python

Made sure I installed latest python 3.9 using asdf

asdf plugin-add python 
asdf install python 3.9.15

Then did,

direnv allow .

That should create a python venv under $PWD/.direnv, then did

pip install -U pip wheel grpcio

IMPORTANT: Python versions other than 3.9.x did not work for my e.g. 3.10 fails with some clang issue

Answer 4

After many attempts, I finally got it installed by running these commands:

xcode-select --install
brew install openssl re2

export LDFLAGS="-L$(/opt/homebrew/bin/brew --prefix openssl)/lib -L$(/opt/homebrew/bin/brew --prefix re2)/lib" 
export CPPFLAGS="-I$(/opt/homebrew/bin/brew --prefix openssl)/include -I$(/opt/homebrew/bin/brew --prefix re2)/include" 

GRPC_BUILD_WITH_BORING_SSL_ASM="" 
GRPC_PYTHON_BUILD_SYSTEM_RE2=true 
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true 
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true 

pip install grpcio

Taken from here: https://candid.technology/error-command-usr-bin-clang-failed-with-exit-code-1/

Answer 5

CFLAGS="-I /opt/homebrew/opt/openssl/include" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt

this works for me in my mac book pro M1 bigsur. Thanks for the solution

Answer 6

This seems to work well, do the following in the terminal

export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

And then install whatever package that you wanted to install, in my case I was trying to install firebase admin and I had to run the following

pip install firebase-admin

Or just run the below command if you just want to install grpcio

pip install grpcio

I found this thread helpful to understand this issue in detail https://github.com/grpc/grpc/issues/25082

Answer 7

I found a solution that works taking from https://github.com/grpc/grpc/issues/25082 with more detail and the removal of certain unnecessary steps. Note that this only works on python version 3.9 as of this post date.

First, install pyenv and pyenv-virtualenv.

brew install pyenv pyenv-virtualenv

Note that you might need to futz with the shell evaluation variable as simply adding eval "$(pyenv init -)" does not seem to work on the Apple M1 computers. Instead, run the following

pyenv init - >> .zshrc
pyenv virtualenv-init - >> .zshrc

Then go to your python project directory and create an .envrc and register it with direnv

echo 'layout pyenv 3.9.1' > .envrc
direnv allow

Next there install your requirements.txt via

GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt

Collecting google-cloud-pubsub
  Using cached google_cloud_pubsub-2.3.0-py2.py3-none-any.whl (210 kB)
Collecting proto-plus>=1.7.1
  Using cached proto_plus-1.17.0-py3-none-any.whl (42 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Using cached grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting libcst>=0.3.10
  Using cached libcst-0.3.17-py3-none-any.whl (507 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Using cached google_api_core-1.26.1-py2.py3-none-any.whl (92 kB)
Collecting protobuf>=3.12.0
  Using cached protobuf-3.15.6-py2.py3-none-any.whl (173 kB)
Collecting grpcio<2.0.0dev,>=1.0.0
  Using cached grpcio-1.36.1.tar.gz (21.5 MB)
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.5.2
  Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting typing-inspect>=0.4.0
  Using cached typing_inspect-0.6.0-py3-none-any.whl (8.1 kB)
Processing /Users/rkmacmini/Library/Caches/pip/wheels/b7/a5/c4/504d913c2a55bb09c607541578ec5f844d1ff33467abe93ba5/PyYAML-5.4.1-cp39-cp39-macosx_11_0_arm64.whl
Collecting typing-extensions>=3.7.4.2
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting six>=1.13.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting google-auth<2.0dev,>=1.21.1
  Using cached google_auth-1.27.1-py2.py3-none-any.whl (136 kB)
Collecting pytz
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Requirement already satisfied: setuptools>=40.3.0 in ./.direnv/python-3.9.1/lib/python3.9/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-pubsub->-r requirements.txt (line 1)) (49.2.1)
Collecting packaging>=14.3
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting mypy-extensions>=0.3.0
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.1-py3-none-any.whl (12 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.6"
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 1.1 MB/s
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Using legacy 'setup.py install' for grpc-google-iam-v1, since package 'wheel' is not installed.
Using legacy 'setup.py install' for grpcio, since package 'wheel' is not installed.
Installing collected packages: six, protobuf, proto-plus, grpcio, googleapis-common-protos, grpc-google-iam-v1, mypy-extensions, typing-extensions, typing-inspect, pyyaml, libcst, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, pytz, pyparsing, packaging, idna, urllib3, certifi, chardet, requests, google-api-core, google-cloud-pubsub
    Running setup.py install for grpcio ... done
    Running setup.py install for grpc-google-iam-v1 ... done
Successfully installed cachetools-4.2.1 certifi-2020.12.5 chardet-4.0.0 google-api-core-1.26.1 google-auth-1.27.1 google-cloud-pubsub-2.3.0 googleapis-common-protos-1.53.0 grpc-google-iam-v1-0.12.3 grpcio-1.36.1 idna-2.10 libcst-0.3.17 mypy-extensions-0.4.3 packaging-20.9 proto-plus-1.17.0 protobuf-3.15.6 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 rsa-4.7.2 six-1.15.0 typing-extensions-3.7.4.3 typing-inspect-0.6.0 urllib3-1.26.4

NOTE: Some references say you need to point your CFLAGS, etc, at the brew installed openssl locations but that has not seemed necessary for me. If you have issue you can try the command with the CFLAGS enabled via

CFLAGS="-I /opt/homebrew/opt/openssl/include" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt

Answer 8

There's a much simpler and more reliable option:

pip uninstall grpcio
conda install grpcio

Answer 9

I manage to get it work without changing any variable in my OS. This seems to be to do with python version issue.. i was using 3.7.12 failed. 3.8.12 failed. Finally, 3.10.2 got this package installed.

Answer 10

None of the suggestion worked for me. What worked is below (assuming you have homebrew installed):

1. Run brew install miniconda (follow through with - conda init "$(basename "${SHELL}")", as instructed.)
2. Either source your ~/.zshrc or kill your terminal and reopen.
3. Run conda create --name myenv python=3.10.
4. Run conda activate myenv.
5. Run pip install grpcio.

This has brew install the correct arm64 conda env in step 1, hence the rest is doable.