AKS Kubernetes job not able to recognise the image tag 'latest'

Question

I am running a single kubernetes job , below is the yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
  namespace: default
spec:
  template:
    spec:
      imagePullSecrets:
      - name: acrsecret
      containers:
      - name: weatherdispatch
        image: navweathercontainer.azurecr.io/weatherdispatch:latest
        imagePullPolicy: Always
      restartPolicy: Never
      nodeSelector:
        beta.kubernetes.io/os: linux
        kubernetes.io/role: agent
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists
      - effect: NoSchedule
        key: azure.com/aci
  backoffLimit: 0

In the image i have mentioned tag as 'latest' like i have done for the kubernetes deployment but this tag is not working and job is throwing the error

Message
containerinstance.ContainerGroupsClient#CreateOrUpdateSender: Failure sending request: StatusCode=400 -- Original Error: Code="InaccessibleImage" Message="The image 'navweathercontainer.azurecr.io/weatherdispatch' in container group 'default-weatherdispatch-job-s4vxl' is not accessible. Please check the image and registry credential."

But when i am giving the image tag value as actual value (like the image tag value as 24939 ) then it is working fine.

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
  namespace: default
spec:
  template:
    spec:
      imagePullSecrets:
      - name: acrsecret
      containers:
      - name: weatherdispatch
        image: navweathercontainer.azurecr.io/weatherdispatch:24939
        imagePullPolicy: Always
      restartPolicy: Never
      nodeSelector:
        beta.kubernetes.io/os: linux
        kubernetes.io/role: agent
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists
      - effect: NoSchedule
        key: azure.com/aci
  backoffLimit: 0

Can someone help me to find the problem when i am using tag 'latest' to pull the image from container registry. Thanks in advance!!

Answer 1

Message containerinstance.ContainerGroupsClient#CreateOrUpdateSender: Failure sending request: StatusCode=400 -- Original Error: Code="InaccessibleImage" Message="The image 'navweathercontainer.azurecr.io/weatherdispatch' in container group 'default-weatherdispatch-job-s4vxl' is not accessible. Please check the image and registry credential."

The error message clearly indicates that you have provided invalid credentials or the container registry password may have expired.

However, you mentioned that using a different tag number works. This could be because the image with that particular tag was used earlier, and Kubernetes stores the image with tag in the local cache of the node where the job is running. Kubernetes utilizes a local cache to store previously pulled images, so if the image is already present in the cache, the job can use it without pulling it from ACR again.

To reproduce this issue, I followed the steps below: I pushed an image to ACR. At this time, the image is having only one tag.

Now I connected to AKS cluster and created a docker-registry secret object to pull the image from ACR and added the imagePullSecrets fiels in jobs manifest file.

apiVersion: batch/v1
kind: Job
metadata:
  name: testjob
spec:
  template:
    spec:
      imagePullSecrets:
      - name: secret-testjob
      containers:
      - name: pi
        image: acrvjy.azurecr.io/testjob:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4

I created a job using the above manifest and it was successfully created and completed.

I then proceeded to test the scenario where an incorrect credential was used by updating the Docker registry secret object. I created the job with the incorrect credential, expecting it to fail. However, the job completed successfully as you see in the below screenshot. When I troubleshooted this issue, I got to know that it did not used imagePullSecrets to pull the image as it is available in the node cache.

Next, I pushed the same image to ACR with the tag "latestt" and updated the code accordingly. When I attempted to create a job using this updated image, I encountered the error messagen as shown in below screenshot.

apiVersion: batch/v1
kind: Job
metadata:
  name: testjob02
spec:
  template:
    spec:
      imagePullSecrets:
      - name: secret-testjob
      containers:
      - name: pi
        image: acrvjy.azurecr.io/testjob:latestt
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4

To ressolve the issue, I updated the container registry credentials and verfied by creating a new jobs with all possible tags.