Nginx upstream routing issue in EKS cluster

Question

I’ve encountered an upstream routing issue with an application served by Nginx. I’ve run out of ideas and would appreciate any insight into what I need to do resolve it.

The application consists of a frontend written in ReactJS, and a backend service written in Python and served by the Uvicorn web server. I use Nginx configured as a reverse proxy server because once I’ve got routing resolved for this application, my intention is to add a second ReactJS frontend and Python backend application.

The apps each run inside their own Kubernetes pod in a single EKS cluster running in a dedicated namespace (called test-namespace) on a single node.

The information in this paragraph may not be relevant but in the interests of full context: build and deployment is done via Skaffold; CI/CD is done by Jenkins; the Skaffold commands are executed from the Jenkinsfile.

% kubectl -n test-namespace get pods
NAME                                     READY   STATUS    RESTARTS   AGE
alb-ingress-controller-6997bd94f-fwcmp   1/1     Running   1          18h
external-dns-5bbd59bf85-hbq8v            1/1     Running   0          18h
backend-api-755bcd5f6-jz9l7              1/1     Running   0          18h
nginx-6c4c6774b4-bwmkp                   1/1     Running   0          18h

% kubectl -n test-namespace get svc
NAME              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
backend-api       ClusterIP      172.20.201.246   <none>        80/TCP         3d23h
nginx             LoadBalancer   172.20.106.196   <pending>     80:31979/TCP   7d21h

% kubectl -n test-namespace get endpoints
NAME              ENDPOINTS         AGE
backend-api       10.3.164.145:80   3d23h
nginx             10.3.184.139:80   7d21h

/etc/nginx/nginx.conf (unmodified):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log info;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf (created by me):

server {
    index index.html;

    listen *:8080;
    listen 0.0.0.0:80;

    # Required for the load balancer health check only.
    location / {
        alias /usr/share/nginx/html/app/;
    }

    # Reverse proxy routes for the ReactJS app.
    location /app1/ {
        alias /usr/share/nginx/html/frontend_app1/build/;
    }
    location /static/ {
        alias /usr/share/nginx/html/frontend_app1/build/static/;
    }

    # This should allow the frontend to call the backend APIs BUT it's not working. 
    location /api {
        proxy_pass http://backend-api;
    }
}

When I run the React app from the browser on my localhost and view the logs of the container running inside the Nginx pod, I get the following log items:

10.3.162.143 - - [12/Mar/2022:15:14:36 +0000] "GET /static/js/2.62dc7698.chunk.js.map HTTP/1.1" 200 739451 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "84.65.41.38"
10.3.162.143 - - [12/Mar/2022:15:14:36 +0000] "GET /static/js/main.569fe7bb.chunk.js.map HTTP/1.1" 200 49743 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "84.65.41.38"
10.3.162.143 - - [12/Mar/2022:15:14:36 +0000] "GET /static/css/main.501a2d76.chunk.css.map HTTP/1.1" 200 4747 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "84.65.41.38"
10.3.162.143 - - [12/Mar/2022:15:14:36 +0000] "GET /api/reports/ HTTP/1.1" 502 559 "https://redacted.com/app1 /" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "84.65.41.38"
2022/03/12 15:14:36 [error] 29#29: *17282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.3.162.143, server: redacted.com, request: "GET /api/reports/ HTTP/1.1", upstream: "http://172.20.201.246:80/api/reports/", host: "redacted.com", referrer: https://redacted.com/app1/

The logs show the browser request being routed through to the static folder, from which the static ReactJS resources are served. They then show the ReactJS app attempting to call an endpoint on the backend service (GET /api/reports/), but this call fails (111: (Connection refused) while connecting to upstream).

When I view the logs of the container running inside the backend-api pod, I find that nothing has been written as a result of the browser request. This indicates that the Nginx pod is unable to send its request to the backend pod.

I’ve tried adding the following to the http block of /etc/nginx/nginx.conf, but it didn’t resolve the issue or alter the Nginx log messages.

upstream backend-api {
    server http://backend-api:80;
    server http://backend-api:8080;
}  

Any observations on errors or omissions in my configuration would be greatly appreciated. Suggestions on how to troubleshoot further and obtain more verbose diagnostic information would also be helpful.

Answer 1

I've fixed this issue. I noticed that Uvicorn was listening on port 8080. When I changed this to 80, the frontend was then able to talk to the backend. I imagine that an alternative solution would be to add a port-mapping.