So I've been trying to customize the nginx configuration of my ingress-nginx controller that is enabled inside my minikube. I thought about following the steps of Custom NGINX template
that's my deployment.yaml file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
gcp-auth-skip-secret: "true"
spec:
serviceAccountName: ingress-nginx
containers:
- name: ingress-nginx-controller
image: registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
volumeMounts:
- mountPath: /etc/nginx/template
name: nginx-template-volume
readOnly: true
volumes:
- name: nginx-template-volume
configMap:
name: nginx-template
items:
- key: nginx.tmpl
path: nginx.tmpl
in the same folder I got my custom nginx.tmpl file
When I apply that deployment the pod never starts with the following errors:
ingress-nginx-controller-99cbbd675-6sqlr 0/1 CrashLoopBackOff 2 (4s ago) 20s
When I describe the pod, here's what I get:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m19s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-99cbbd675-6sqlr to minikube
Normal Pulled 48s (x5 over 2m18s) kubelet Container image "registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
Normal Created 48s (x5 over 2m18s) kubelet Created container ingress-nginx-controller
Normal Started 48s (x5 over 2m18s) kubelet Started container ingress-nginx-controller
Warning BackOff 47s (x10 over 2m16s) kubelet Back-off restarting failed container ingress-nginx-controller in pod ingress-nginx-controller-99cbbd675-6sqlr_ingress-nginx(0c542d9b-a052-457c-8d92-225db6df9287)
Full description of deployment:
Name: ingress-nginx-controller-99cbbd675-6sqlr
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: minikube/192.168.49.2
Start Time: Tue, 20 Feb 2024 13:09:05 +0100
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
gcp-auth-skip-secret=true
pod-template-hash=99cbbd675
Annotations: <none>
Status: Running
IP: 10.244.1.234
IPs:
IP: 10.244.1.234
Controlled By: ReplicaSet/ingress-nginx-controller-99cbbd675
Containers:
ingress-nginx-controller:
Container ID: docker://1fb60b233cfe467fddd2eaac5453754f886c19d10118cadb0a560c4bc8bccd7a
Image: registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 20 Feb 2024 13:10:36 +0100
Finished: Tue, 20 Feb 2024 13:10:36 +0100
Ready: False
Restart Count: 4
Environment: <none>
Mounts:
/etc/nginx/template from nginx-template-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6qbmn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nginx-template-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-template
Optional: false
kube-api-access-6qbmn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m19s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-99cbbd675-6sqlr to minikube
Normal Pulled 48s (x5 over 2m18s) kubelet Container image "registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
Normal Created 48s (x5 over 2m18s) kubelet Created container ingress-nginx-controller
Normal Started 48s (x5 over 2m18s) kubelet Started container ingress-nginx-controller
Warning BackOff 47s (x10 over 2m16s) kubelet Back-off restarting failed container ingress-nginx-controller in pod ingress-nginx-controller-99cbbd675-6sqlr_ingress-nginx(0c542d9b-a052-457c-8d92-225db6df9287)
Edit: I checked back the pod logs
$ kubectl logs -n ingress-nginx ingress-nginx-controller-6674b884ff-2bmf9
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.9.4
Build: 846d251814a09d8a5d8d28e2e604bfc7749bcb49
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
-------------------------------------------------------------------------------
W0221 09:13:39.037739 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0221 09:13:39.037821 7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0221 09:13:39.041852 7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.3" state="clean" commit="a8a1abc25cad87333840cd7d54be2efaf31a3177" platform="linux/amd64"
I0221 09:13:39.286149 7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
F0221 09:13:39.288020 7 main.go:120] Unexpected error obtaining ingress-nginx pod: unable to get POD information (missing POD_NAME or POD_NAMESPACE environment variable
Tried adding POD_NAMESPACE and POD_NAME like this
containers:
- name: ingress-nginx-controller
image: registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- mountPath: /etc/nginx/template
name: nginx-template-volume
readOnly: true
Checked back the logs:
$ kubectl log -n ingress-nginx ingress-nginx-controller-5fd667fc7d-lnvpl
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.9.4
Build: 846d251814a09d8a5d8d28e2e604bfc7749bcb49
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
-------------------------------------------------------------------------------
W0221 09:30:36.001546 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0221 09:30:36.001625 7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0221 09:30:36.005473 7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.3" state="clean" commit="a8a1abc25cad87333840cd7d54be2efaf31a3177" platform="linux/amd64"
I0221 09:30:36.100787 7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
E0221 09:30:36.109126 7 store.go:871] unexpected error parsing name and ns: invalid format (namespace/name) found in ''
W0221 09:30:36.109149 7 store.go:875] Unexpected error reading configuration configmap: resource name may not be empty
F0221 09:30:36.110646 7 nginx.go:172] Invalid NGINX configuration template: template: nginx.tmpl:424: function "builds" not defined
This is the configmap is created using this command:
kubectl create configmap nginx-config --from-file=ingress-nginx/nginx.tmpl -n ingress-nginx
nginx.tmpl is the same template provided by the official k8s' image just added some headers inside the http block
Help is really appreciated!