Install multiple private packages from Github using Poetry and deploy keys

Question

I want to install multiple private Github repositories as python packages into my repository using poetry. This works fine when running poetry install locally as I have my public SSH key added to Github which allows poetry to access the private repos. The problem is that I want to install these same private packages in my CI/CD pipeline and for that I need to add one deploy key per repo. The correct deploy key needs to be used towards the correct repository and for that to work, I need to setup a config with some aliases on the following format (I haven't actually gotten so far as to know whether this will actually work or not):

// /.ssh/config
Host repo-1.github.com
 IdentityFile ~/.ssh/repo-1-deploy-key
Host repo-2.github.com
 IdentityFile ~/.ssh/repo-2-deploy-key

Where repo-1 and repo-2 are the names of the private repositories I need to install. When running locally the pyproject.toml packages need to be setup in the following format:

// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://[email protected]/equinor/repo-1.git" }
repo-2 = { git = "ssh://[email protected]/equinor/repo-2.git" }
...

As this will allow the developer to install the packages without any configuration (given that they have access). For the CI/CD pipeline however, the URLs need to match the alias in the SSH config file and therefore it needs to look something like this:

// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://[email protected]/equinor/repo-1.git" }
repo-2 = { git = "ssh://[email protected]/equinor/repo-2.git" }
...

Now, where I seem to be stuck at is on how I can include two different git paths in the same pyproject file? I tried the following:

//pyproject.toml

[tool.poetry.dependencies]
repo-1 = { git = "ssh://[email protected]/equinor/repo-1.git", optional=true }
repo-2 = { git = "ssh://[email protected]/equinor/repo-2.git", optional=true }

[tool.poetry.dev-dependencies]
repo-1 = { git = "ssh://[email protected]/equinor/repo-1.git" }
repo-2 = { git = "ssh://[email protected]/equinor/repo-2.git" }

[tool.poetry.extras]
cicd_modules = ["repo-1", "repo-2"]

So that I could run poetry install locally and it would use the dev dependencies and poetry install --no-dev --extras cicd_modules in the CI/CD pipeline to use the aliased paths. Sadly, this gives me a CalledProcessError as it seems like poetry tries to install the optional package despite the optional flag set to true.

What am I doing wrong here, am I using the optional flag incorrectly somehow? Is there a better way to solve this? In summary I just want to be able to install multiple private repositories as packages using poetry and Github deploy keys in a CI/CD pipeline without breaking local install behaviour, if it's in this way or another better way, I don't really have a strong opinion on.

Answer 1

If you are running in the CI/CD pipeline a subtly different configuration, consider using a shell script to modify the text of the pyproject.toml file. In this case I might use a sed one-liner to replace ssh://[email protected] with https://PRIVATE_TOKEN:[email protected].

sed -i "s/ssh://[email protected]/https://$CI_CD_USER:[email protected]/g" pyproject.toml

And add the appropriate environment variable values for CI_CD_USER and CI_CD_TOKEN to your configuration.

Answer 2

I think the problem here is that you are inventing new hosts (e.g., repo-1.github.com) but those hostnames don't actually exist, and you are not mapping them to the correct existing hostname (github.com).

Your .ssh/config should look like:

Host repo-1.github.com
 Hostname github.com
 IdentityFile ~/.ssh/repo-1-deploy-key
Host repo-2.github.com
 Hostname github.com
 IdentityFile ~/.ssh/repo-2-deploy-keyHost repo-0

Then, you can use repo-1.github.com and repo-2.github.com as if they were real hosts, and they will be routed to github.com:

...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://[email protected]/equinor/repo-1.git" }
repo-2 = { git = "ssh://[email protected]/equinor/repo-2.git" }
...