Google Cloud Build: pip install from JFrog private repository using Cloud KMS as secretEnv

Question

I am trying do pip install private packages in jfrog repository through Google Cloud Build. I can access jfrog repository via https:<USER>:<API_KEY>@<artifactory_url> and can also do pip install <package_name_and_version> https:<USER>:<API_KEY>@<artifactory_url>

I want to integrate this step in cloud build using Cloud KMS to decrypt API_KEY during pip install. I have encrypted API_KEY string using the following command

# Create a local file with the secret
echo "MyAPIKEY" > plain_pwd.txt
# To encrypt a secret using KMS
gcloud kms encrypt \
  --plaintext-file=plain_pwd.txt \
  --ciphertext-file=cipher_pwd.enc.txt \
  --location=global \
  --keyring=<keyring> \
  --key=<key>
# Encode the binary encoded secret as base64 string
base64 cipher_pwd.enc.txt -w 0 > cipher_pw.enc.64.txt

specified secrets in cloudbuild.yaml as:

secrets:
- kmsKeyName: projects/<project_id>/locations/global/keyRings/<keyring>/cryptoKeys/<key>
  secretEnv:
    APIKEY: <base64 encrypted string from cloud kms encrypt command>

This is my cloudbuild.yaml step for pip install:

-
    args:
      - "-m"
      - pip
      - install
      - "-t"
      - /workspace/lib
      - "schema-registry-client==0.8.14.dev0"
      - "--extra-index-url"
      - 'https://adminuser:$$APIKEY@<artifactory_url>'
    entrypoint: python3
    secretEnv: ['APIKEY']
    id: INSTALL_SCHEMA_REGISTRY
    name: "python:3.7"

I get 'EOFError: EOF when reading a line' in cloud build while executing this step:

Looking in indexes: https://pypi.org/simple,

https://onpat:****@artifactory.build.****.****.com/artifactory/api/pypi/dpfw-pypi-dev-local/simple
 ERROR: Exception:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
     status = self.run(options, args)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
     return func(self, options, args)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 325, in run
     reqs, check_supported_wheels=not options.target_dir
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
     discovered_reqs.extend(self._resolve_one(requirement_set, req))
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
     abstract_dist = self._get_abstract_dist_for(req_to_install)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
     self._populate_link(req)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
     req.link = self._find_requirement_link(req)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
     best_candidate = self.finder.find_requirement(req, upgrade)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 899, in find_requirement
     req.name, specifier=req.specifier, hashes=hashes,
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 881, in find_best_candidate
     candidates = self.find_all_candidates(project_name)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 826, in find_all_candidates
     project_url, link_evaluator=link_evaluator,
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 790, in process_project_url
     html_page = self._link_collector.fetch_page(project_url)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 643, in fetch_page
     return _get_html_page(location, session=self.session)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 455, in _get_html_page
     resp = _get_html_response(url, session=session)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 169, in _get_html_response
     "Cache-Control": "max-age=0",
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
     return self.request('GET', url, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/session.py", line 421, in request
     return super(PipSession, self).request(method, url, *args, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 650, in send
     r = dispatch_hook('response', hooks, r, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
     _hook_data = hook(hook_data, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
     username, password, save = self._prompt_for_password(parsed.netloc)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
     username = ask_input("User for {}: ".format(netloc))
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
     return input(message)
 EOFError: EOF when reading a line
Finished Step #2 - "INSTALL_SCHEMA_REGISTRY"
ERROR
ERROR: build step 2 "python:3.7" failed: step exited with non-zero status: 2

Also, I tried gcloud kms decrypt on same ciphertext and I get the original API KEY back. So, I don't think encrypt/decrypt is an issue here. I have also given necessary access to cloud build service agents to cloud kms.

Any suggestions/help on how to fix this?

Answer 1

I have found an alternate way to solve this problem, Apparently, pip cli has an issue while expanding arguments and env variables. You can use Google Cloud Secret manager to store the credentials (JFrog API_KEY in this case). Use gcloud secrets versions access <version_id> --secret=<secret_key> to retrieve the secret at runtime. Create pip.conf file at ~/.pip/pip.conf with --extra-index-url=https://user:<api_key_secret_value>@<artifactory_url> in the build step and just do pip install <package_name> in the subsequent build step. Only downside is it adds one extra step in the build process. I created bash scripts to read the secret to a file in build step and create pip.conf value using that secret value in next build step. Use cloud build volumes to pass files between steps.

#read_secret.sh

#!/bin/bash
gcloud secrets versions access 1 --secret=artifactory-api-key > "/data/key.txt"

install the package in the next step with the following bash script

#install_module.sh

#!/bin/bash
export API_KEY=$(cat "/data/key.txt")
echo $API_KEY
mkdir -p ~/.pip
echo -e "[global]\n--extra-index-url=https://myuser:$API_KEY@<artifactory_url>" > ~/.pip/pip.conf
pip install -t "/workspace/lib" private-package-lib==0.1.2