Install a private package with credentials from requirements.txt

Question

I want to a private package via

%pip install -r /dbfs/requirements.txt

Inside this requirements.txt is give:

git+https://<user>:$token@<gitprovider>.com/<path/to/repo>

If i give user and token inside the requirements.txt it works like a charm, but I want to pass user and token via dbutils.secrets. Is this doable without manipulating the requirements.txt before?

I thought it would be possible to create environment variables, but this did not work for me.

%env GIT_USERNAME= ...
%env GIT_TOKEN= ...

Answer 1

The way is using python

1. Getting information
2. Store the information in a temporary file so that its details will not be found in the run log or records after running
3. Run

import dbutils

username = dbutils.secrets.get(scope="your_scope", key="git_username")
token = dbutils.secrets.get(scope="your_scope", key="git_token")

repo_url = f"git+https://{username}:{token}@gitprovider.com/path/to/repo"

requirements_path = "/dbfs/tmp/requirements.txt"

with open(requirements_path, "w") as file:
    file.write(repo_url + "\n")

dbutils.notebook.run("%pip install -r " + requirements_path, 60)

Answer 2

I believe the simplest way would be to use a ~/.netrc file, that way you don't pollute your requirements.txt with usernames/passwords.

If you've never used a ~/.netrc file, then touch ~/.netrc first, otherwise add this line (either programmatically or manually)

machine <gitprovider>.com login <user> password <token>

Now your requests should pick up the authentication without you having to manually specify it, i.e. this is what your requirement.txt line is

git+https://<gitprovider>.com/<path/to/repo>

Note: If this is running in a pipeline, etc. you can also use a ~/.netrc, and clear it up when the pipeline ends or rely on whatever sort of garbage collection exists.