What is the node hardware requirements and kubernetes nginx ingress configuration required processing large files (3gb) in AKS cluster?

Question

We have a requirement to process collection of files up to the limit of 3gb size in total via .net core api pods hosted in AKS cluster.

The 3 nodes cluster has the current node configuration as 16gb Ram and 2vCpus. I have also added some ingress annotations as below:

nginx.ingress.kubernetes.io/client-body-buffer-size: 5000M
nginx.ingress.kubernetes.io/client-max-body-size: 5000M
nginx.ingress.kubernetes.io/proxy-body-size: 5000M
nginx.ingress.kubernetes.io/proxy-connect-timeout: '14400'
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: '14400'
nginx.ingress.kubernetes.io/proxy-read-timeout: '14400'
nginx.ingress.kubernetes.io/proxy-send-timeout: '14400'
nginx.ingress.kubernetes.io/server-snippet: keepalive_timeout 3600s; client_body_timeout 3600s;

These changes helped to handle up to 2gb file sizes, by ingress utilizing 1.02cpu and 1000mi memory for almost 5 minutes. The latest resource requirement I've updated for ingress is as below, but it didn't help either.

resources:
    requests:
      cpu: 1400m
      memory: 3000Mi

But struggling to process sizes more than 2gb and gets terminated after processing around 7 minutes and throwing an internal server error.

I couldn't find the exact log message yet and any suggestions on configuring azure metrics, logs, alerts is also helpful in addition to hardware and ingress configuration to handle 3gb file sizes successfully.

Answer 1

For uploads of that size either publish the service port directly, or if the ingress provides some other functionality, disable request buffering in nginx.

Put the upload service on it's own ingress host: upload.x.com

  nginx.ingress.kubernetes.io/proxy-request-buffering: off
  nginx.ingress.kubernetes.io/proxy-body-size: 3G

Failing that, buffer requests to temporary files so nginx isn't keeping the requests in memory.

  nginx.ingress.kubernetes.io/configuration-snippet: |
    client_body_in_file_only: on
  nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 3G
  nginx.ingress.kubernetes.io/client-body-buffer-size: 64k
  nginx.ingress.kubernetes.io/proxy-body-size: 3G