Cannot install apache-beam==2.35.0, apache-beam[gcp]==2.35.0 apache-beam[gcp]==2.47.0 and apache-beam[gcp]==2.48.0 versions have conflict dependencis

Question

I have a kfp component that installs apache-beam==2.35.0. I run it in Vertex AI on GCP pipelines.

Component header:

@component(
    base_image="dockerhub/python:3.7.12",
    packages_to_install=["tensorflow-data-validation==1.6.0", "apache-beam==2.35.0"]
)
def generate_statistics(
    statistics: Output[Artifact],
    dataset: str,
) -> None:
   
...

But I'm getting this error: error screenshot

ERROR: Cannot install apache-beam==2.35.0, apache-beam[gcp]==2.35.0, apache-beam[gcp]==2.39.0, apache-beam[gcp]==2.45.0, apache-beam[gcp]==2.47.0 and apache-beam[gcp]==2.48.0 because these package versions have conflicting dependencies.

I don't know why it tries to install multiple versions at the same time?!

Answer 1

tensorflow-data-validation library has a dependency on apache-beam and so it installs it.

So unless you have requirement to use specific version of apache-beam, remove that from the requiquirement and you should be good.