Unable to install pypi package(google-cloud-bigquery) on GCP Composer

Question

I am trying to install google-cloud-bigquery==1.5.0 PyPI packages on GCP composer for a new environment recently has been created. I get this error:

Successfully installed google-cloud-bigquery-1.5.0 google-cloud-core-0.28.1 pypd-1.1.0 strict-rfc3339-0.7
+ [[ -z fail ]]
+ python3 -m pipdeptree --warn fail
Warning!!! Possibly conflicting dependencies found:
* google-cloud-translate==2.0.1
 - google-cloud-core [required: >=1.1.0,<2.0dev, installed: 0.28.1]
* google-cloud-storage==1.29.0

I tried another version (2.2.0) and it had conflicts with some other google pre-installed packages. The new environment image version is composer-1.12.2-airflow-1.10.6. There is another environment created few months ago and all pypi packages are installed successfully and airflow dags are running smoothly, the image version for it is composer-1.10.0-airflow-1.10.6 .

Question 1: I think the current issue is linked to the image version and probably I have to recreate the new environment with older image version, am I correct?

Question 2: To create new environment I have only three options for image version which are composer-1.12.2-airflow-1.10.6, composer-1.12.2-airflow-1.10.9, composer-1.12.2-airflow-1.10.10. How I can create environment with image version composer-1.10.0 ? We have several other projects and environment with the same location/zone but with composer-1.10.0

Answer 1

Please, have a look to the official documentation for the Apache Airflow and Python versions that Cloud Composer supports. You can refer to the section for the Python packages which comes with composer-1.12.2-airflow-1.10.6. A specific Cloud Composer Airflow version already comes with a set of included packages. When you upgrade/downgrade a specific installed PyPi package or try to use any other packages, you can run into conflicts.

As for now, there is no way to check conflicts ahead of time within Cloud Composer environment. I would like to suggest you to adjust your packages to be able to use in the one of the Composer environments: composer-1.12.4-airflow-1.10.10, composer-1.12.4-airflow-1.10.9 or composer-1.12.4-airflow-1.10.6.