I have a Kubernetes cluster running in AWS, and I am working through upgrading various components. Internally, we are using NGINX, and it is currently at v1.1.1 of the nginx-ingress chart (as served from old stable), with the following configuration:
controller:
publishService:
enabled: "true"
replicaCount: 3
service:
annotations:
external-dns.alpha.kubernetes.io/hostname: '*.MY.TOP.LEVEL.DOMAIN'
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: [SNIP]
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
targetPorts:
http: http
https: http
My service's ingress resource looks like...
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
[SNIP]
spec:
rules:
- host: MY-SERVICE.MY.TOP.LEVEL.DOMAIN
http:
paths:
- backend:
serviceName: MY-SERVICE
servicePort: 80
path: /
status:
loadBalancer:
ingress:
- hostname: [SNIP]
This configuration works just fine, however, when I upgrade to v3.11.1 of the ingress-nginx chart (as served from the k8s museum).
With an unmodified config, curling to the HTTPS scheme redirects back to itself:
curl -v https://MY-SERVICE.MY.TOP.LEVEL.DOMAIN/INTERNAL/ROUTE
* Trying W.X.Y.Z...
* TCP_NODELAY set
* Connected to MY-SERVICE.MY.TOP.LEVEL.DOMAIN (W.X.Y.Z) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.MY.TOP.LEVEL.DOMAIN
* start date: Aug 21 00:00:00 2020 GMT
* expire date: Sep 20 12:00:00 2021 GMT
* subjectAltName: host "MY-SERVICE.MY.TOP.LEVEL.DOMAIN" matched cert's "*.MY.TOP.LEVEL.DOMAIN"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> GET INTERNAL/ROUTE HTTP/1.1
> Host: MY-SERVICE.MY.TOP.LEVEL.DOMAIN
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Content-Type: text/html
< Date: Wed, 28 Apr 2021 19:07:57 GMT
< Location: https://MY-SERVICE.MY.TOP.LEVEL.DOMAIN/INTERNAL/ROUTE
< Content-Length: 164
< Connection: keep-alive
<
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host MY-SERVICE.MY.TOP.LEVEL.DOMAIN left intact
* Closing connection 0
(I wish I had captured more verbose output...)
I tried modifying the NGINX config to append the following:
config:
use-forwarded-headers: "true"
and then...
config:
compute-full-forwarded-for: "true"
use-forwarded-headers: "true"
These did not seem to make a difference. It was in the middle of the day, so I wasn't able to dive too far in before rolling back.
What should I look at, and how should I debug this?
Update:
I wish that I had posted a complete copy of the updated config, because I would have noticed that I did not correctly apply the change to add config.compute-full-forwarded-for: "true". It need to be within the controller block, and I had placed it elsewhere.
Once the compute-full-forwarded-for: "true" config was added, everything started to work immediately.
This is a community wiki answer posted for better visibility. Feel free to expand it.
As confirmed by @object88 the issue was with misplaced
config.compute-full-forwarded-for: "true"configuration which was located in the wrong block. Adding it to thecontrollerblock solved the issue.