AKS Ingress with Service Type "ExternalName" to Azure Static website. Getting "ErrorFetchingEndpoints"

Question

To utilize Azure static website as the backend on ingress, we attempted to utilize the service type as ExternalName. However, this approach proved ineffective, resulting in the following errors:

Warning  EndpointsEmpty  9m17s (x58 over 17m)  azure/application-gateway  Code="ErrorFetchingEndpoints" Message="Endpoint not found for nodepool/xxxblobbackend"

We aim to route static traffic based on different paths, thereby alleviating the load on the primary workload.

We have successfully deployed the application on AKS with Azure Application Gateway as the ingress, and it is performing as anticipated.

Presently, we endeavor to redirect the static content to the Azure Blob static page instead of routing it through our application. Initially, we manually deployed it on the application gateway, which yielded the desired outcome. However, according to Microsoft Documentation, making manual changes on Application Gateway post-integration with AKS is not recommended.

We have also attempted to apply it through the ingress controller, as outlined below, but we got a negative outcome as mentioned above.

apiVersion: v1
kind: Service
metadata:
  name: xxxblobbackend
  namespace: nodepool
spec:
  type: ExternalName
  externalName: xxxyyyprod.z16.web.core.windows.net
  selector:
    app: xxxblobbackend
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    annotations:
      appgw.ingress.kubernetes.io/backend-hostname: localhost
      appgw.ingress.kubernetes.io/backend-protocol: http
      appgw.ingress.kubernetes.io/health-probe-path: /
      appgw.ingress.kubernetes.io/request-timeout: "180"
      appgw.ingress.kubernetes.io/proxy-body-size: "0"
      appgw.ingress.kubernetes.io/proxy-read-timeout: "900"
      appgw.ingress.kubernetes.io/proxy-send-timeout: "900"
      appgw.ingress.kubernetes.io/ssl-redirect: "true"
      kubernetes.io/ingress.class: azure/application-gateway
    generation: 5
    name: ingress-api
    namespace: nodepool
spec:
  rules:
    - http:
        paths:
        - path: /
          pathType: ImplementationSpecific
          backend:
            service:
              name: defaultbackend
              port:
                number: 80
        - path: /api/*
          pathType: Prefix
          backend:
            service:
              name: api-service
              port:
                number: 8081
        - path: /static/*
          pathType: ImplementationSpecific
          backend:
            service:
              name: xxxblobbackend
              port:
                number: 443
  tls:
  - secretName: xxxme-yyy-tls-certificate

Answer 1

When integrating an Azure Blob Storage static website with AKS using Azure Application Gateway as the ingress and encountering the "ErrorFetchingEndpoints" issue with a Service of type ExternalName, a direct approach with ExternalName might not work as expected due to the way Azure Application Gateway Ingress Controller (AGIC) handles service endpoints. Instead, consider using an Nginx reverse proxy within your cluster to route traffic to the static website. This method bypasses the limitations of ExternalName services with AGIC.

First, deploy an Nginx pod that acts as a reverse proxy. This pod will forward requests to your Azure Blob Storage static website.

Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-static-proxy
  namespace: nodepool
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-static-proxy
  template:
    metadata:
      labels:
        app: nginx-static-proxy
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-conf
          mountPath: /etc/nginx/conf.d/static-site.conf
          subPath: static-site.conf
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-static-proxy-config

ConfigMap for Nginx Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-static-proxy-config
  namespace: nodepool
data:
  static-site.conf: |
    server {
      listen       80;
      
      # Specify a resolver
      resolver 8.8.8.8 valid=10s; # Use Kubernetes cluster DNS or a public DNS resolver
      set $upstream http://xxxyyyprod.z16.web.core.windows.net;
      
      location / {
        proxy_pass $upstream;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
      }
    }

In this configuration, resolver 8.8.8.8 valid=10s; tells Nginx to use 8.8.8.8 (Google's DNS) as the resolver with a validity time of 10 seconds for cached DNS lookups. You can replace 8.8.8.8 with the IP address of your Kubernetes cluster's DNS service if you prefer not to rely on an external service.

Next, expose the Nginx reverse proxy through a Kubernetes service within your cluster.

service yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-static-proxy-service
  namespace: nodepool
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx-static-proxy

Finally, adjust your Ingress resource to route /static/* paths to the Nginx proxy service. Finally, adjust your Ingress resource to route /static/* paths to the Nginx proxy service.

ingress yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-api
  namespace: nodepool
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    # Other necessary annotations...
spec:
  rules:
    # Your other rules...
    - http:
        paths:
        - path: /static/*
          pathType: Prefix
          backend:
            service:
              name: nginx-static-proxy-service
              port:
                number: 80
  tls:
  - secretName: xxxme-yyy-tls-certificate

The original issue was that AGIC could not properly route to the Azure Blob Storage static site using a service of type ExternalName, resulting in "ErrorFetchingEndpoints" errors. This is because AGIC expects to route traffic to IP endpoints, which ExternalName services do not provide directly.

Here, in this setup, Nginx acts as a middleman, receiving requests from the ingress and forwarding them to the Azure Blob Storage static site and should route traffic intended for static content to the Azure Blob Storage through an Nginx reverse proxy, circumventing the limitations of ExternalName with AGIC and enabling efficient static content delivery.