Pulling a image from gcr.to fails

Question

I am able to create a kubernetes cluster and I followed the steps in to pull a private image from GCR repository. https://cloud.google.com/container-registry/docs/advanced-authentication https://cloud.google.com/container-registry/docs/access-control

I am unable to pull the image from GCR. I have used the below commands gcloud auth login I have authendiacted the service accounts. Connection between the local machine and gcr as well.

Below is the error

$ kubectl describe pod test-service-55cc8f947d-5frkl
Name:         test-service-55cc8f947d-5frkl
Namespace:    default
Priority:     0
Node:         gke-test-gke-clus-test-node-poo-c97a8611-91g2/10.128.0.7
Start Time:   Mon, 12 Oct 2020 10:01:55 +0530
Labels:       app=test-service
              pod-template-hash=55cc8f947d
              tier=test-service
Annotations:  kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container test-service
Status:       Pending
IP:           10.48.0.33
IPs:
  IP:           10.48.0.33
Controlled By:  ReplicaSet/test-service-55cc8f947d
Containers:
  test-service:
    Container ID:   
    Image:          gcr.io/test-256004/test-service:v2
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      test_SERVICE_BUCKET:   test-pt-prod
      COPY_FILES_DOCKER_IMAGE:  gcr.io/test-256004/test-gcs-copy:latest
      test_GCP_PROJECT:      test-256004
      PIXALATE_GCS_DATASET:     test_pixalate
      PIXALATE_BQ_TABLE:        pixalate
      APP_ADS_TXT_GCS_DATASET:  test_appadstxt
      APP_ADS_TXT_BQ_TABLE:     appadstxt
    Mounts:
      /test/output from test-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6g7nl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  test-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  test-pvc
    ReadOnly:   false
  default-token-6g7nl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6g7nl
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                From                                                          Message
  ----     ------                  ----               ----                                                          -------
  Normal   Scheduled               42s                default-scheduler                                             Successfully assigned default/test-service-55cc8f947d-5frkl to gke-test-gke-clus-test-node-poo-c97a8611-91g2
  Normal   SuccessfulAttachVolume  38s                attachdetach-controller                                       AttachVolume.Attach succeeded for volume "pvc-25025b4c-2e89-4400-8e0e-335298632e74"
  Normal   SandboxChanged          31s                kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 15s (x2 over 32s)  kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Pulling image "gcr.io/test-256004/test-service:v2"
  Warning  Failed                  15s (x2 over 32s)  kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Failed to pull image "gcr.io/test-256004/test-service:v2": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/test-256004/test-service, repository does not exist or may require 'docker login': denied: Permission denied for "v2" from request "/v2/test-256004/test-service/manifests/v2".
  Warning  Failed                  15s (x2 over 32s)  kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Error: ErrImagePull
  Normal   BackOff                 3s (x4 over 29s)   kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Back-off pulling image "gcr.io/test-256004/test-service:v2"
  Warning  Failed                  3s (x4 over 29s)   kubelet, gke-test-gke-clus-test-node-poo-c97a8611-91g2  Error: ImagePullBackOff

Answer 1

If you don't use workload identity, the default service account of your pod is this one of the nodes, and the nodes, by default, use the Compute Engine service account.

Make sure to grant it the correct permission to access to GCR.

If you use another service account, grant it with the Storage Object Reader role (when you pull an image, you read a blob stored in Cloud Storage (at least it's the same permission)).

Note: even if it's the default service account, I don't recommend to use the Compute Engine service account with any change in its roles. Indeed, it is project editor, that is a lot of responsability.