Stop Nomad from exposing services to the Internet

Question

I have Nomad installed on a VPS. When I run a task nomad creates a docker container which is directly exposed to the public internet.

Current Behaviour

However if I try to reach it on 127.0.0.1:<port> or 0.0.0.0:<port> it is not available. It is available from inside the VPS and outside using <public_ip>:<port>.

Expected Behaviour

It cannot be reached from the internet using <public_ip>:<port> but it can be reached from inside the VPS using 127.0.0.1:<port> or 0.0.0.0:<port>

Nomad Config File

data_dir  = "/root/nomad/data"
bind_addr = "0.0.0.0"

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
  servers = ["127.0.0.1"]
}

acl {
  enabled = true
}

Job File

job "echo-app" {
  datacenters = ["dc1"]

  group "web" {
    count = 1

    network {
      port "appHttp" {
        to = 8080
      }
    }

    service {
      provider = "nomad"
      port     = "appHttp"
      name     = "echo-web-http"
    }

    task "echo-server" {
      driver = "docker"
      config {
        image          = "mendhak/http-https-echo:26"
        ports          = [
          "appHttp"
        ]
      }

      resources {
        cpu    = 100
        memory = 200
      }
    }

  }

}

How my network interfaces look

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:4e:2e:6f brd ff:ff:ff:ff:ff:ff
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:34:7a:1d:69:e9 brd ff:ff:ff:ff:ff:ff
7: veth0ad3ded@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 12:d7:56:da:bc:5f brd ff:ff:ff:ff:ff:ff link-netnsid 0

Answer 1

Stop Nomad from exposing services to the Internet

To stop Nomad, or anything else, from exposing service to the Internet configure firewall on your server. Firewall is the basic for a public VPS.

Note that docker installs rules in nat PREROUTING chain which can bypass firewall configurations. See https://docs.docker.com/network/packet-filtering-firewalls/ . Research integration between your favorite firewall manager and docker. I think that firewall-cmd and shorewall have support for docker.

if I try to reach it on 127.0.0.1:

Yes. In Nomad client the default interface is the one with the default gateway, i.e. eth0 in your case. So Nomad is not redirecting from 127.0.0.1.

or 0.0.0.0:

Connecting to 0.0.0.0 IP port is highly odd. Use localhost when you want to connect to the local host. Use 0.0.0.0 for listening only.

it can be reached from inside the VPS using 127.0.0.1:

Add localhost to your nomad client configuration. See https://developer.hashicorp.com/nomad/docs/configuration/client#host_network-block . On my private server I have:

client {
  network_interface       = "lo"
  host_network "default" {
    interface = "lo"
  }
  host_network "lo" {
    interface = "lo"
  }
  host_network "ext" {
    interface = "eth0"
  }

}

Then your job will use the default lo interface, or explicitly specify lo interface in your job. See https://developer.hashicorp.com/nomad/docs/job-specification/network#host_network .

network {
  port "appHttp" {
    to = 8080
    host_nerwork = "lo"  // or leave at default
  }
}

or 0.0.0.0:

Once again, use localhost.

If you are wondering if you can make nomad service listen on 0.0.0.0 IP address, there is no such option. In that case, use network_mode = "host" and make the service listen on 0.0.0.0 and leave a like on this issue https://github.com/hashicorp/nomad/issues/12106 .