nginx & nodejs: connect() failed (111: Connection refused) while connecting to upstream

Question

Before I get into it, I have looked through every similar issue I could find and nothing I found solved my problem.

I am running 2 docker containers, 1 for nginx and 1 for a nodejs api. I am using nginx as a reverse proxy.

When I request localhost/api/x, I receive a 502 bad gateway, and nginx logs

nginx_1 | 2015/08/15 15:30:30 [error] 9#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: X.X.X.X, server: _, request: "GET /api/x HTTP/1.1", upstream: "http://127.0.0.1:8543/x", host: "localhost"

I am able to get the port that 8543 is mapped to on my computer with docker ps, and can access the application there just fine. That leads me to the conclusion that the node app is running, and listening on the correct port (8543). I am also able to tell from the error message, and from the rewrite log, that the request uri is getting rewritten properly.

I have been banging my head against this for a while, and can't figure out what is wrong, any help would be appreciated!

My nginx.conf:

user nginx nginx;

worker_processes auto;
worker_rlimit_nofile 8192;

events {
  worker_connections 8000;
}

error_log  /dev/stderr notice;
pid        /var/run/nginx.pid;

http {

  # Hide nginx version
  server_tokens off;

  rewrite_log on;

  # mime types
  include mime.types;
  default_type application/octet-stream;

  # Update charset_types due to updated mime.types
  charset_types text/xml text/plain text/vnd.wap.wml application/x-javascript application/rss+xml text/css application/javascript application/json;

  # Format to use in log files
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log /dev/stdout main;

  keepalive_timeout 20;
  sendfile          on;
  tcp_nopush        on;

  # Nodejs API
  upstream api {
    server 127.0.0.1:8543;
  }

  # Reverse-proxy for the Riot API, S3, and our API
  server {
    listen [::]:80;
    listen 80;

    server_name _;

    charset utf-8;

    # Resolver
    resolver         8.8.8.8 valid=300s;
    resolver_timeout 10s;

    # API, reverse proxy our API
    location /api/ {
      limit_except GET {
        deny all;
      }
      rewrite ^/api(/.*)$ $1 break;
      proxy_pass http://api;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
    }
  }
}

My nginx Dockerfile

# Set the base image to centos
FROM centos:7

# File Author
MAINTAINER Andrew Shapro

# Install nginx
RUN yum update -y && \
  yum -y install \
  curl \
  tar \
  openssl-devel \
  gcc \
  gcc-c++ \
  make \
  zlib-devel \
  pcre-devel \
  gd-devel \
  krb5-devel

# download and compile nginx
RUN curl -sLo nginx-1.9.2.tar.gz http://nginx.org/download/nginx-1.9.2.tar.gz \
  && mkdir /root/nginx_source \
  && tar -xzvf nginx-1.9.2.tar.gz -C /root/nginx_source --strip-components=1
COPY nginx_modules /root/nginx_modules
RUN cd /root/nginx_source \
  && ./configure \
    --user=nginx \
    --with-debug \
    --group=nginx \
    # --prefix=/user/share/nginx \
    # --sbin-path=/user/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --pid-path=/run/nginx.pid \
    --http-client-body-temp-path=/var/lib/nginx/body \
    --http-proxy-temp-path=/var/lib/nginx/proxy \
    --lock-path=/run/lock/subsys/nginx \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_auth_request_module \
    --with-http_addition_module \
    --with-http_gzip_static_module \
    --with-http_image_filter_module \
    --with-http_spdy_module \
    --with-http_sub_module \
    --with-ipv6 \
    --add-module=/root/nginx_modules/ngx_devel_kit \
    --add-module=/root/nginx_modules/ngx_aws_auth \
  && make \
  && make install \
  && rm -rf /root/nginx_modules /root/nginx_source \
  && mkdir --parents /var/lib/nginx

# Add nginx configs
RUN curl -sLo /usr/local/bin/ep https://github.com/kreuzwerker/envplate/releases/download/v0.0.8/ep-linux && chmod +x /usr/local/bin/ep
ADD nginx.conf /etc/nginx/nginx.conf
ADD mime.types /etc/nginx/mime.types

# Cleanup after build
RUN yum clean all \
  && yum autoremove -y

# Add nginx user
RUN adduser -c "Nginx user" nginx

EXPOSE 80

# Run nginx
# CMD ["stat", "/usr/local/nginx/sbin/nginx"]
CMD [ "/usr/local/bin/ep", "-v", "/etc/nginx/nginx.conf", "--", "/usr/local/nginx/sbin/nginx", "-c", "/etc/nginx/nginx.conf", "-g", "daemon off;" ]

My node app

var express = require('express');

var app = express();

function allTheRoutes (req, res) {
  res.send('Hello World!');
}

app.get('/', allTheRoutes);
app.get('/*', allTheRoutes);
var app = require('./src/app');
var server = app.listen(process.env.PORT || 8543, process.env.HOST || '127.0.0.1', function () {
    var host = server.address().address;
    var port = server.address().port;

    console.log('Listening at http://%s:%s', host, port);
});

My node Dockerfile

# Set the base image to centos
FROM centos:7

# File Author
MAINTAINER Andrew Shapro

# Install nginx
RUN yum update -y && \
  yum -y install curl

# Add nvm to $PATH
ENV PATH $PATH:$HOME/.nvm/bin

# Install nvm
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.26.0/install.sh | bash

# Clean up
RUN yum clean all \
  && yum autoremove -y

# Add iojs to $PATH
ENV PATH $PATH:/root/.nvm/versions/io.js/v3.0.0/bin

# Intall iojs
RUN bash -lic 'nvm install iojs-v3.0'

# Install PM2
RUN npm install -g pm2

# NPM install
WORKDIR /app
ADD package.json /app/
RUN npm install

# Add app
ADD . /app

# Add run.sh
ADD run.sh run.sh
RUN chmod 755 run.sh

# Expose port
EXPOSE 8543

# run node
CMD ["./run.sh"]

./run.sh runs the command pm2 start -x app.js --no-daemon

docker-compose.yml

nginx:
    build: nginx
    ports:
      - "80:80"
    link: node
  node:
    build: node
    ports:
      - "8543"
    environment:
      - HOST=127.0.0.1
      - PORT=8543

Answer 1

From your docker-compose.yml you are exposing port 80 from the nginx container to port 80 outside the container, but you are exposing port 8543 to a random port. When you tell nginx to look for 127.0.0.1:8543 it won't find that port in the host.

You are already linking both containers together, you don't need to expose port 8543 to the host in order to access if from nginx. All you need to do is tell it to access the right host:

# Nodejs API
upstream api {
    server node:8543;
}

This happens because when you link containers docker set the host in /etc/hosts. You can check it:

docker exec -ti nginx bash
cat /etc/hosts

Of course, it should work if you expose 8543:

nginx:
build: nginx
ports:
  - "80:80"
link: node
node:
build: node
ports:
  - "8543:8543"
environment:
  - HOST=127.0.0.1
  - PORT=8543

More on docker networking: https://docs.docker.com/articles/networking/