How to setup ingress for Azure Kubernetes cluster using AGIC?

Question

I have two applications that I want to deploy to my cluster, and thought that I could prefix them in the ingress with different paths to reach them.

When I setup the first application using the below setup everything works as expected, I can reach the application using the public IP of the application gateway and navigate it:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Exact
        backend:
          service:
            name: store-front
            port:
              number: 80

But when I try to add my second application in the mix I want to update the first application to use a specific path, for example /store.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /store
        pathType: Exact
        backend:
          service:
            name: store-front
            port:
              number: 80

My problem now is that only the first request works, so going to <PUBLIC_IP>/store gives me the page but all sub sequent requests from the application are now wrong, since they do not use the prefix of /store. So all css and js requests break.

So my question is if there is a way to handle this in the ingress and not in the application (as if I don't have access to modify the routing in the app)?

Can I setup the ingress to make/force all sub sequent requests from the application somehow use the /store prefix?

I have also tried this (that I saw was suggested when others have asked similar questions) without any luck getting it registered in my application gateway backend pool.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - http:
      paths:
      - path: /store(/|$)(.*)
        pathType: Exact
        backend:
          service:
            name: store-front
            port:
              number: 80

I assume this does not work since it is not the appgw.ingress annotation and thus not supported.

Answer 1

I find your statement that only the first request works, which pretty much looks like "health probe down" issue.

There is a statement in official document as below:

If the above probes aren't provided, then the Ingress Controller makes an assumption that the service is reachable on the Path specified for backend-path-prefix annotation, or the path specified in the ingress definition for the service.

See also: Add Health Probes to your service

Since you have appgw.ingress.kubernetes.io/backend-path-prefix: "/", I suggest you make sure there will be 200/ok response code returned from the path /.
In simple: set up any normal page behind the path /, and even blank page will work, as long as it returns 200/ok response code.
Note: cannot be the return code like 404.

Also, do check out "health probe" in application gateway to verify the root cause. See also: Application Gateway health probes overview