Kubernetes can't find locally built containerd images

Question

I'm running an on-premise K8s cluster. The problem is Kubernetes doesn't see the locally built container images, despite the images being visible in the k8s.io namespace - tested both with nerdctl --namespace=k8s.io images and crictl images commands. crictl guide

Cluster information:

Kubernetes version: 1.26.1

on-premise cluster

Host OS: AlmaLinux 9.1

CNI and version: Calico v3.24.5

Container Runtime: Containerd v1.6.16

[root@alma-kube ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
alma-kube    Ready    control-plane   62d   v1.26.1
alma-kube2   Ready    worker          61d   v1.26.0

crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
...
docker.io/library/mytestimage             latest              f240b96f676e1       2.81MB
docker.io/library/nginx-test              1                   5e34a0811eefb       56.9MB
docker.io/library/nginx-test              latest              5e34a0811eefb       56.9MB
docker.io/nick/nginx                      latest              5e34a0811eefb       56.9MB
docker.io/registry-nick/nginx-test        latest              5e34a0811eefb       56.9MB

nerdctl --namespace=k8s.io images
REPOSITORY                                 TAG        IMAGE ID        CREATED         PLATFORM       SIZE         BLOB SIZE
...
mytestimage                                latest     dfb4f2a8f7c9    5 hours ago     linux/amd64    5.6 MiB      2.7 MiB
nginx-test                                 1          7f7e7eb6b3c2    2 days ago      linux/amd64    146.7 MiB    54.3 MiB
registry                                   latest     3f71055ad7c4    2 days ago      linux/amd64    23.7 MiB     8.8 MiB
registry-nick/nginx-test                   latest     7f7e7eb6b3c2    3 days ago      linux/amd64    146.7 MiB    54.3 MiB

The Pod YAML file:

Note: The commented out "#image:" lines are things I tried and all of them failed with the same [error](### The pod fails to start with ErrImageNeverPull error:). I tried accessing the image directly - this is my primary goal. Because that didn't work, I also tried by running a private registry.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-test-pod
spec:
  containers:
  - name: nginx-test-crp
    image: mytestimage
    #image: docker.io/library/nginx-test:1
    #image: registry-nick/nginx-test
    #image: localhost:5000/nginx-test
    imagePullPolicy: Never

I create the pod using the command kubectl apply -f pod-test.yaml

The pod fails to start with ErrImageNeverPull error:

kubectl get pod -o wide --all-namespaces
default            nginx-test-pod                             0/1     ErrImageNeverPull   0               124m   192.168.48.11    alma-kube2   <none>  

kubectl logs nginx-test-pod
Error from server (BadRequest): container "nginx-test-crp" in pod "nginx-test-pod" is waiting to start: ErrImageNeverPull

kubectl describe pod nginx-test-pod
Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  ErrImageNeverPull  58s (x630 over 136m)  kubelet  Container image "local-nick/bokluk" is not present with pull policy of Never

Obviously K8s can't see the locally built image. How can I fix this problem?

Answer 1

You need to transfer the image onto the worker node.

A local registry will not work while you have the pod configured with imagePullPolicy: Never.

If you want to keep your container images on-premises then either:

• set up a local image registry server and relax the image pull policy, or
• pre-load the image onto your worker nodes (perhaps using their launch scripts).