I'm migrating from ingress-nginx-controller to aws-loadbalancer-controller to provision access to some UIs running in my K8S cluster via their ingresses.
My aim is to use a single ALB that will route requests for ArgoCD, Grafana, Prometheus and AlertManager UIs. So far this has worked for the ArgoCD and Grafana UIs, and the IPs of the AlertManager and Prometheus pods are being successfully registered into their respective target groups and are healthy, but requesting the prometheus URL in a browser returns a 404 for the /graph redirect. I can however successfully curl -L http://prom-pod-ip:9090 from another pod running in the cluster and the redirect to /graph works fine when I do that.
The alertmanager UI request fails with a 404 for script.js.
Prometheus ingress config is:
ingress:
enabled: true
annotations:
#kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/group.name: internal-support
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: {{ .Values.internalIngress.sslCertArn }}
alb.ingress.kubernetes.io/healthcheck-path: "/graph"
alb.ingress.kubernetes.io/healthcheck-port: "traffic-port"
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '10'
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
alb.ingress.kubernetes.io/healthy-threshold-count: '2'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
alb.ingress.kubernetes.io/tags: {{ .Values.internalIngress.resourceTags }}
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
hosts:
- demo.product.company.com
As you can see above, i'm actually using /graph for the target group's health check path, and since the prometheus pod's IP is being successfully registered, the target group itself has no issue getting a 200 response from it directly. Requesting https://demo.product.company.com/graph (not the genuine host name) in a browser doesn't work though and gets a 404.
I've seen other posts suggesting tweaks to hosts e.g. to use demo.product.company.com/* or to add externalUrl config, but none of that has helped.