nginx conf uwsgi_pass not respecting DNS TTL

Question

I am using nginx along with an upstream uwsgi app. The uwsgi app is using AWS Cloudmap to serve as service discovery.

The problem I have encountered is that cloudmap uses DNS resolution to the uwsgi hosts. It seems that uwsgi resolves and caches whatever goes into uwsgi_pass.

e.g. in uwsgi conf I have the line uwsgi_pass uwsgi://web.sandbox:8000;

This means that the nginx server works fine until a box in the DNS gets deleted or swapped out. Then I get an error saying nginx cannot connect because it's trying to go to the old nonexistent box.

No route to host) while connecting to upstream, client: 12.151.32.34, server: sandbox.mydomain.com, request: "GET /member/api/user/ HTTP/1.1", upstream: "uwsgi://172.30.1.89:8000"

I tried using resolver_timeout 0s; but that appears to not affect uwsgi_pass.

The DNS TTL is set to 10 seconds so nginx is not respecting that.

How do I get uwsgi_pass to either respect the TTL or to resolve the domain every time?

Answer 1

The solution I found is to add the following to the top of my nginx conf

    # these two lines force DNS resolution (so if ip changes due to LB is replacement or if we use DNS service discovery)
    resolver ${DNS_SERVER};
    set $uwsgi_django_backend "${UWSGI_SERVER_FQDN}";

replace the two variables with ones appropriate for your environment.

Now you can do proxy_pass $uwsgi_django_backend; and it will re-evaluate the DNS.

The DNS resolver is usually just 127.0.0.1 but sometimes you'll need to find it manually (like if you're in a special containerized environment)

export DNS_SERVER=$(cat /etc/resolv.conf |grep -i '^nameserver'|head -n1|cut -d ' ' -f2)