Pulling images from private google container registry with kubeflow on minikube

Question

We are having trouble giving a container within a pipeline uploaded to Kubeflow access to a private custom docker image stored in a google container registry. We are running kubeflow on top of a kubernetes cluster run on minikube. Can someone help us understand how to add the access token/service account to the Kubeflow deployment? We have read a couple of docs that achieve this on a custom Kubernetes deployment but not on a Kubeflow deployment.

The error we get when running the pipeline on Kubeflow is: This step is in Pending state with this message: ImagePullBackOff: Back-off pulling image

This is the pipeline code that calls the image.

Thank you!!

Answer 1

This is issues can occur in some scenarios like:

• Your kubeflow setup (Kubernetes cluster) and GCR are in different project

• No GCR secret for the ml-pipeline service account which is responsible to run the pipeline. (you can see this kubectl --namespace=kubeflow get serviceaccount)

In your case, I think it is the second scenario. Though the following path will work on both scenarios.

1. Create service_account.json with sufficient permission (GCR needs storage permission so give 'Storage admin') using the GCP console

Select “API & Services” > “Credentials”Select “Create credentials” > “Services Account Key” > “Create New Services Account”

2. Add a Kubernetes Secret in Kubernetes Cluster to access GCR

kubectl create secret docker-registry $SECRETNAME \       
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
[email protected] \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key

• Above method is for default service account. But patch this in Kufelow namespace

kubectl --namespace=kubeflow create secret docker-registry $SECRETNAME \  
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
[email protected] \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key

3. Patching GCR secret with respective service account

# For Kubeflow specific problem path pipeline-runner serviceaccount
kubectl --namespace=kubeflow patch serviceaccount pipeline-runner -p '{"imagePullSecrets": [{"name": "$SECRETNAME"}]}'