Can't install private Artifactory Cloud PyPi package within Cloud Composer environment

Question

I've set up a Python package in a private instance of Artifactory Cloud and am trying to add it as a dependency to my composer environment using the following steps:

1. Adding the following pip.conf file to the config folder of the composer gcs bucket

[global]
extra-index-url = https://<USERNAME>:<TOKEN>@<ENTERPRISE>.jfrog.io/artifactory/api/pypi/<REPO_NAME>/simple/

2. Granting the composer service account the roles artifactregistry.reader and compute.viewer
3. Adding the package to the Composer environment's PyPi packages using the Composer UI However, I get the following error:

+ PYTHON=python3
+ echo 'Installing Python3 Requirements.'
+ python3 -m pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://<USERNAME>:****@<ENTERPRISE>.jfrog.io/artifactory/api/pypi/<REPO_NAME>/simple/
Requirement already satisfied: google-api-python-client>=1.6.0 in /opt/python3.8/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (2.107.0)
Requirement already satisfied: google-auth-httplib2>=0.1.0 in /opt/python3.8/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (0.1.1)
Requirement already satisfied: google-auth-oauthlib>=0.4.1 in /opt/python3.8/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (1.0.0)
Requirement already satisfied: pandas>=1.5.3 in /opt/python3.8/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (2.0.3)
Collecting weasyprint>=59.0
  Downloading weasyprint-61.2-py3-none-any.whl (271 kB)
ERROR: Could not find a version that satisfies the requirement <REPO_NAME> (from -r requirements.txt (line 6)) (from versions: none)
ERROR: No matching distribution found for <REPO_NAME> (from -r requirements.txt (line 6))
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION  fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

Which is strange since it picks up the index but doesn't find any packages inside. The same pip.conf file allows me to install the package locally so I'm thinking it might have to do something with the Composer service account permissions, but none of the permissions I've tried have worked.

Answer 1

I've seen this issue before it had something to do with how I was specifying the "extra-index-url" IIRC.

looking at the error "Looking in indexes:" it is trying to find the pkg in https://pypi.org/simple repo I think that's the problem here as the env is private env it should not have access to public internet. PTAL [1]

[1] https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-private-only