nginx auth_request fails with 'upstream timed out'

Question

I'm writing a simple reverse proxy to exchange a JWT for an API while connecting to a VOD service (livepeer). Testing on my machine, using docker.

But the request always hangs. Here's the nginx.conf:

events {
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    access_log  /dev/stdout;
    error_log  /dev/stdout;

    server {

        listen 8080;

        server_name creators_proxy;

        location /api {
            auth_request /auth;
            proxy_set_header Authorization 'Bearer here_goes_the_key';
            proxy_pass https://livepeer.studio/api;
        }

        location = /auth {
            internal;
            proxy_pass here_goes_auth_url;
        }

        location = /auth2 {
            proxy_pass here_goes_auth_url;
        }
    }
}

and the dockerfile:

# Use the official NGINX image as a parent image
FROM nginx:latest

COPY nginx.conf /etc/nginx/nginx.conf

I build the container with docker run -d --name docker-nginx -p 8080:8080 -v ./nginx.conf:/etc/nginx/nginx.conf -d nginx_img

And call http://localhost:8080/api/asset/request-upload

The auth_url is from pipedream, to inspect the request going out. But the request never lands. I get:

upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.17.0.1, server: creators_proxy, request: "POST /api/asset/request-upload HTTP/1.1", subrequest: "/auth", upstream: "https://xx.xxx.xx.xx:443/auth", host: "localhost:8080"

What I've tried so far:

1. curl'd the auth url from my machine (it works)
2. curl'd the auth url from the docker instance (it works)
3. Called location /auth2 to go directly to the auth_url (it works)
4. Checked whether this nginx version has the auth_request module enabled (it has)
5. Commented out the auth_requestin the /api location (doesn't hang)

Any ideas?

Answer 1

I was able to make it work by adding two directives to the /auth location:

location = /auth {
    internal;
    proxy_pass              http://auth-server;
    proxy_pass_request_body off;
    proxy_set_header        Content-Length "";
}

This comes from the docs where it's said that:

As the request body is discarded for authentication subrequests, you will need to set the proxy_pass_request_body directive to off and also set the Content-Length header to a null string

My mistake was to think the /auth location as being special because it was called from the auth_request module.

Answer 2

Your setup involves using Pipedream as a stand-in for the actual authentication endpoint:

+---------------------+
|    Client           |
|  (localhost:8080)   |
+----------+----------+
           |
           v
+----------+----------+
|    NGINX Reverse    |
|    Proxy (Docker)   |
|   - Port 8080       |
|   - auth_request    |
+----------+----------+
           |
           v
+----------+----------+
|    Auth Endpoint    |
|     (Pipedream)     |
+---------------------+

Given this setup, you need to make sure the proxy_pass directive in your NGINX configuration under the /auth location is correctly pointing to the Pipedream URL. An incorrect URL means failed authentication requests.

Also, make sure the Docker container can reach the Pipedream URL (acting as your auth endpoint):

docker network ls
docker network inspect [network_name]

docker exec -it [container_name_or_id] /bin/bash

# Inside the container, run:
ping -c 4 google.com
curl [pipedream_url]

nslookup livepeer.studio
nslookup [pipedream_domain]

cat /etc/resolv.conf

Do replace [xxx] placeholder values with their actual values.

The default timeouts might be too short. Try and update your nginx.conf with increased timeout settings and debug logging for some clues:

events {
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    access_log  /dev/stdout;
    error_log  /dev/stdout debug;  # Enable debug logging

    server {

        listen 8080;

        server_name creators_proxy;

        location /api {
            auth_request /auth;
            proxy_set_header Authorization 'Bearer here_goes_the_key';
            proxy_pass https://livepeer.studio/api;

            proxy_read_timeout 300s;  # Increased timeout
            proxy_connect_timeout 300s;
        }

        location = /auth {
            internal;
            proxy_pass here_goes_pipedream_url;  # Make sure this points to Pipedream

            proxy_read_timeout 300s;
            proxy_connect_timeout 300s;
        }

        location = /auth2 {
            proxy_pass here_goes_pipedream_url;
        }
    }
}

Do replace here_goes_pipedream_url with the correct URL for your Pipedream endpoint.

Keep an eye on the logs/output on Pipedream to check if it receives requests from NGINX. If Pipedream is slow in responding, or if there are issues with the response reaching NGINX, it could lead to timeouts. That is particularly important, since you have increased the timeout settings.