Currently I have a cluster with single controller and single worker, also a nginx reverse-proxy (hhtp only) outside cluster.
Controller is at 192.168.1.65 worker is at 192.168.1.61 reverse proxy at 192.168.1.93 and public ip
here is my ingress-nginx services
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.102.58.7 192.168.1.186 80:31097/TCP,443:31116/TCP 56m
ingress-nginx-controller-admission ClusterIP 10.108.233.49 <none> 443/TCP 56m
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.3.0
Annotations: <none>
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.102.58.7
IPs: 10.102.58.7
LoadBalancer Ingress: 192.168.1.186
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31097/TCP
Endpoints: 10.244.0.23:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31116/TCP
Endpoints: 10.244.0.23:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
that 192.168.1.186 is assigned by MetalLB.
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl get IPAddressPools -A
NAMESPACE NAME AGE
metallb-system pool01 99m
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n metallb-system describe IPAddressPool pool01
Name: pool01
Namespace: metallb-system
Labels: <none>
Annotations: <none>
API Version: metallb.io/v1beta1
Kind: IPAddressPool
Metadata:
Creation Timestamp: 2022-07-26T09:08:10Z
Generation: 1
Managed Fields:
API Version: metallb.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:addresses:
f:autoAssign:
f:avoidBuggyIPs:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-26T09:08:10Z
Resource Version: 41021
UID: 2a0dcfb2-bf8f-4b1a-b459-380e78959586
Spec:
Addresses:
192.168.1.186 - 192.168.1.191
Auto Assign: true
Avoid Buggy I Ps: false
Events: <none>
I deploy hello-app at namespace : 'dev'
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev get all
NAME READY STATUS RESTARTS AGE
pod/hello-app-5c554f556c-v2gx9 1/1 Running 1 (20m ago) 63m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-service ClusterIP 10.111.161.2 <none> 8081/TCP 62m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-app 1/1 1 1 63m
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-app-5c554f556c 1 1 1 63m
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev describe service hello-service
Name: hello-service
Namespace: dev
Labels: app=hello
Annotations: <none>
Selector: app=hello
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.111.161.2
IPs: 10.111.161.2
Port: <unset> 8081/TCP
TargetPort: 8080/TCP
Endpoints: 10.244.0.22:8080
Session Affinity: None
Events: <none>
Local tests of that service:
bino@k8s-worker-1:~$ curl http://10.111.161.2:8081
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
bino@k8s-worker-1:~$ curl http://10.244.0.22:8080
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
and the ingress resource of that service:
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev describe ingress hello-app-ingress
Name: hello-app-ingress
Labels: <none>
Namespace: dev
Address: 192.168.1.61
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
bino.k8s.jcamp.cloud
/ hello-service:8081 (10.244.0.22:8080)
Annotations: ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 23m (x3 over 24m) nginx-ingress-controller Scheduled for sync
When I open http://bino.k8s.jcamp.cloud I got 502
my nginx reverse proxy conf :
server {
listen 80 default_server;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://192.168.1.186;
}
}
The nginx error log say
2022/07/26 06:24:21 [error] 1593#1593: *6 connect() failed (113: No route to host) while connecting to upstream, client: 203.161.185.210, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://192.168.1.186:80/favicon.ico", host: "bino.k8s.jcamp.cloud", referrer: "http://bino.k8s.jcamp.cloud/"
from describe ingress-nginx-controller pod
bino@corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx describe pod ingress-nginx-controller-6dc865cd86-9fmsk
Name: ingress-nginx-controller-6dc865cd86-9fmsk
Namespace: ingress-nginx
Priority: 0
Node: k8s-worker-1/192.168.1.61
Start Time: Tue, 26 Jul 2022 16:11:05 +0700
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=6dc865cd86
Annotations: kubernetes.io/psp: 00-k0s-privileged
Status: Running
IP: 10.244.0.23
IPs:
IP: 10.244.0.23
Controlled By: ReplicaSet/ingress-nginx-controller-6dc865cd86
Containers:
controller:
Container ID: containerd://541446c98b55312376aba4744891baa325dca26410abe5f94707d270d378d881
Image: registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Image ID: registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Tue, 26 Jul 2022 16:56:40 +0700
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Tue, 26 Jul 2022 16:11:09 +0700
Finished: Tue, 26 Jul 2022 16:56:26 +0700
Ready: True
Restart Count: 1
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6dc865cd86-9fmsk (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfmrc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-nfmrc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning NodeNotReady 44m node-controller Node is not ready
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "webhook-cert" : object "ingress-nginx"/"ingress-nginx-admission" not registered
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "webhook-cert" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "kube-api-access-nfmrc" : failed to sync configmap cache: timed out waiting for the condition
Normal SandboxChanged 43m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 43m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5" already present on machine
Normal Created 43m kubelet Created container controller
Normal Started 43m kubelet Started container controller
Warning Unhealthy 42m (x2 over 42m) kubelet Liveness probe failed: Get "http://10.244.0.23:10254/healthz": dial tcp 10.244.0.23:10254: connect: connection refused
Warning Unhealthy 42m (x3 over 43m) kubelet Readiness probe failed: Get "http://10.244.0.23:10254/healthz": dial tcp 10.244.0.23:10254: connect: connection refused
Normal RELOAD 42m nginx-ingress-controller NGINX reload triggered due to a change in configuration
and here is the nft ruleset
bino@k8s-worker-1:~$ su -
Password:
root@k8s-worker-1:~# systemctl status nftables.service
● nftables.service - nftables
Loaded: loaded (/lib/systemd/system/nftables.service; enabled; vendor preset: enabled)
Active: active (exited) since Tue 2022-07-26 05:56:17 EDT; 46min ago
Docs: man:nft(8)
http://wiki.nftables.org
Process: 186 ExecStart=/usr/sbin/nft -f /etc/nftables.conf (code=exited, status=0/SUCCESS)
Main PID: 186 (code=exited, status=0/SUCCESS)
CPU: 34ms
Warning: journal has been rotated since unit was started, output may be incomplete.
[]
Complete ruleset is at https://pastebin.com/xd58rcQp
Kindly please tell me what to do, to check, or to learn for fixing this problem
Sincerely -bino-
my bad ... There is a name mismatch between ip pool devinition yaml and the l2 advertisement yaml.