Use SSH key of host during Singularity/Apptainer build

Question

When building a Singularity/Apptainer image from a definition file, is there a portable way to make a SSH key of the host system available during the build?

To give some context:

I have a definition file where in the %post section I'm cloning a private git repository using SSH, i.e.:

git clone [email protected]:luator/private_repo.git

This fails because the SSH keys of the host system are not available in the container during the build.

I could probably copy the key in the container and delete it from there at the end of the build process. However, for this, I would need to hard-code the path to the key in the definition file, which is bad when using the same definition file on another machine where the path is different. Is there a more portable way of making the git clone work during the build?

Answer 1

This answer is based on discussions with drdaved on the Apptainer Slack.

The way we found to do this was to let the Apptainer build process use the host machine's ssh-agent by setting the SSH_AUTH_SOCK environment variable. That way, if your key is known to the agent on the host, it can be used in the container during build.

The SSH_AUTH_SOCK on the host must be bound into the container. Check where your SSH_AUTH_SOCK is on the host; if it's in a directory that is already bound into the container during build (e.g. /tmp), you're ready to go. Otherwise, you need to bind it in manually. A common location for the socket is something like /run/user/1000/keyring/ssh, so you could add --bind /run into the container build command.

The advantage of this method is that the key is at no point copied to the container image. The only information that is stored is the socket that your host machine's ssh-agent was running on. AFAIK this is not a security issue.

Note that we have to do some mild trickery as environment variables on the host are not passed to the build process, and there is no way of directly setting environment variables in the build command. Hence, we set via --build-arg, and use the build arg in %post to set the environment variable.

Example test.def file:

Bootstrap: docker
From: ubuntu:latest

%post
    apt update
    apt install git -y

    export SSH_AUTH_SOCK={{ SSH_AUTH_SOCK }}
    git clone [email protected]:luator/private-repo.git

Build command (if your SSH_AUTH_SOCK will already bound to the container):

apptainer build --build-arg SSH_AUTH_SOCK=$SSH_AUTH_SOCK test.sif test.def

Build command (if your SSH_AUTH_SOCK will not already bound to the container, and is available at /run/...):

apptainer build --build-arg SSH_AUTH_SOCK=$SSH_AUTH_SOCK --bind /run test.sif test.def

Answer 2

You would need a method that does not require hardcoding the path to the key. That involves mounting the SSH key as a temporary file within the container during the build.

Before starting the build, read your SSH key into an environment variable, in a shell on the host system:

export SSH_KEY=$(cat /path/to/your/ssh/key)

In your Singularity definition file, use the %setup section to create a temporary file inside the build environment and write the SSH key to it. Since %setup runs on the host system, but can affect the build environment, you can use it to pass the key.

%setup
    echo "$SSH_KEY" > $SINGULARITY_ROOTFS/tmp/ssh_key

In the %post section, move the SSH key to a proper location, set the right permissions, and configure SSH to use it.

%post
    mkdir -p $HOME/.ssh
    mv /tmp/ssh_key $HOME/.ssh/id_rsa
    chmod 600 $HOME/.ssh/id_rsa
    echo "Host github.com\n\tStrictHostKeyChecking no\n\tIdentityFile ~/.ssh/id_rsa" > $HOME/.ssh/config
    git clone [email protected]:luator/private_repo.git
    rm -f $HOME/.ssh/id_rsa

Now, build your Singularity image as usual. The SSH key will be used for the git clone operation and then removed.

That would make sure your SSH key is only used temporarily during the build and does not get hard-coded into the image. Do unset the SSH_KEY environment variable or close the terminal session to prevent the key from remaining in the environment.

Note there is a potential concern: if the image is saved or cached during the build process (especially after the %post section where the key is still present), there might be a risk of the key being recoverable from these intermediate layers or snapshots. That risk needs to be assessed based on how Singularity/Apptainer handles intermediate layers during the build process.

Still, this approach makes sure the same .def file can be used on different machines. The only requirement is to set the SSH_KEY environment variable appropriately on each host before initiating the build.

Answer 3

You can try using docker Buildkit, export DOCKER_BUILDKIT=1 to enable this feature:

And afterwards just generate your ssh keys (ssh-keygen and be sure that your public key - id_rsa.pub file content is in your github/gitlab/bitbucket)

Simple usage in a Dockerfile:

# use you base image
FROM centos AS build
  
RUM yum install -y git

RUN mkdir -m 700 /root/.ssh; \
    touch -m 600 /root/.ssh/known_hosts; \
    ssh-keyscan github.com > /root/.ssh/known_hosts

# update with tour repo
RUN --mount=type=ssh,id=github git clone [email protected]:<USER>/<REPO>.git

And then build the image RUN --mount=type=ssh,id=github git clone [email protected]:<USER>/<REPO>.git