DNS configuration for accessing consul remotely

Question

I have installed consul on AWS EC2, with 3 servers and 1 client.

server IPs = 11.XX.XX.1,11.XX.XX.2,11.XX.XX.3. client IP = 11.XX.XX.4

consul config: /etc/consul.d/server/config.json

{
    "bootstrap": false,
    "server": true,
    "datacenter": "abc",
    "advertise_addr": "11.XX.XX.1",
    "data_dir": "/var/consul",
    "log_level": "INFO",
    "enable_syslog": true,
    "addresses": {
    "http": "0.0.0.0"
    },
    "start_join": ["11.XX.XX.2", "11.XX.XX.3"]
}

netstat output on server:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      29720/consul    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1006/sshd       
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      29720/consul    
tcp6       0      0 :::8301                 :::*                    LISTEN      29720/consul    
tcp6       0      0 :::8302                 :::*                    LISTEN      29720/consul    
tcp6       0      0 :::8500                 :::*                    LISTEN      29720/consul    
tcp6       0      0 :::22                   :::*                    LISTEN      1006/sshd       
tcp6       0      0 :::8300                 :::*                    LISTEN      29720/consul 

curl is working fine from remote machine but dig is only working on the local machine.

; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40873
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;web.service.consul.        IN  A

;; ANSWER SECTION:
web.service.consul. 0   IN  A   11.XX.XX.4

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Dec 30 08:21:41 UTC 2016
;; MSG SIZE  rcvd: 52

but dig is not working from remote machine:

dig @11.XX.XX.1 -p 8600 web.service.consul

; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> @11.XX.XX.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
-----------------------------

How to make it work?

Answer 1

By default consul only listens for DNS connections on the instance loopback device. Best practices asks you to install the client on any remote machine looking to consume consul DNS. This is not always practical.

I have seen people expose DNS (consul port 8600) on all interfaces via the Consul configuration JSON like so:

{
   "server": true,
   "addresses": {
     "dns": "0.0.0.0"
   }
}

You can also expose all ports listening on loopback with the client_addr field in JSON or pass it via the command line with:

consul agent -client 0.0.0.0

There are more controls and knobs available to tweak (see docs):

https://www.consul.io/docs/agent/options.html