Kubernets ingress-nginx timeout in 120s

Question

I have one api, which will run about 4 minutes. I have deployed it on kubernetes with ingress-nginx. All api work normally except the long-run api, it always return 504 Gateaway as below:

I have check in stackoverflow, and try some solution, none of it work for me.

any help is welcome.

Kubernetes Ingress (Specific APP) 504 Gateway Time-Out with 60 seconds

I have changed ingress config as below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{  .Values.releaseName }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
    nginx.ingress.kubernetes.io/server-snippet: "keepalive_timeout 3600s; grpc_read_timeout 3600s; grpc_send_timeout 3600s;client_body_timeout 3600s;"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3601"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3601"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3601"
    # nginx.org/proxy-connect-timeout: 3600s
    # nginx.org/proxy-read-timeout: 3600s
    # nginx.org/proxy-send-timeout: 3600s
    nginx.ingress.kubernetes.io/proxy-body-size: "100M"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error non_idempotent  http_502 http_503 http_504"
    nginx.ingress.kubernetes.io/retry-non-idempotent: "true"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "1"
    # nginx.ingress.kubernetes.io/configuration-snippet: |-
    #   location  /HouseKeeping/Health/Healthz {
    #       deny all;
    #       return 403;
    #     }
    #   location  /InternalApi/ {
    #     deny all;
    #     return 403;
    #   }
    nginx.ingress.kubernetes.io/server-snippets: |
      http {
        client_max_body_size 100m;
      }
      location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Port  $server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection $http_connection;
        proxy_cache_bypass $http_upgrade;
        proxy-connect-timeout: 3602;
        proxy-read-timeout: 3602;
        proxy-send-timeout: 3602;
      }
      
spec:
  rules:
    - host: {{  .Values.apiDomain }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{  .Values.releaseName }}-web-clusterip-srv
                port:
                  number: 80

I also change the configMap for ingress-nginx-controller to add below config:

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  proxy-connect-timeout: "300"
  proxy-read-timeout: "300"
  proxy-send-timeout: "300"
kind: ConfigMap

I also used command to get ingress-nginx's conf, it seems okay

kubectl -n ingress-nginx exec ingress-nginx-controller-6cc65c646d-ljmrm cat /etc/nginx/nginx.conf | tee nginx.test-ingress-export.conf

 # Custom headers to proxied server

 proxy_connect_timeout                   3601s;
 proxy_send_timeout                      3601s;
 proxy_read_timeout                      3601s;

it still timeout in 120 seconds.

Answer 1

Typo

Replace below (ingress config):

nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "1"

In place of (ingress config)

nginx.ingress.kubernetes.io/proxy_next_upstream_timeout: "5"
nginx.ingress.kubernetes.io/proxy_next_upstream_tries: "1"

Edit :

Check this comment may help to resolve your issue.