I am unable to form a ES cluster between 2 master nodes in EC2 instances. Following is the elasticsearch.yml for the nodes.
Node1:
bootstrap.memory_lock: true
cloud.aws.protocol: http
cloud.aws.proxy.host: <Proxy addr>
cloud.aws.proxy.port: <proxy port>
cloud.aws.region: us-east
cluster.name: production-test
discovery.ec2.availability_zones: us-east-1a,us-east-1b,us-east-1d,us-east-1e
discovery.zen.ping_timeout: 30s
discovery.ec2.tag.Name: <ec2-tag name>
discovery.zen.hosts_provider: ec2
#discovery.type: ec2
#discovery.zen.ping.multicast.enabled: false
http.port: 9205
#network.host: _eth0_, _local_, _ec2_
network.host: <private ip_addr>
#network.bind_host: <private ip_addr>
#network.publish_host: <private ip_addr>
node.data: true
node.master: true
plugin.mandatory: discovery-ec2, repository-s3
transport.tcp.port: 9305
#discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>","<private ip_addr of node2>"]
discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>:9305", "<private ip_addr of node2>:9305"]
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
node.name: nodetest1
path.data: /var/lib/elasticsearch/
#path.data: /data/elasticsearch/data/production
path.logs: /var/log/elasticsearch/
path.conf: /etc/elasticsearch
Node 2:
bootstrap.memory_lock: true
cloud.aws.protocol: http
cloud.aws.proxy.host: <Proxy addr>
cloud.aws.proxy.port: <Procy port>
cloud.aws.region: us-east
cluster.name: production-test
discovery.ec2.availability_zones: us-east-1a,us-east-1b,us-east-1d,us-east-1e
discovery.zen.ping_timeout: 30s
discovery.ec2.tag.Name: <ec2-instance tag name>
discovery.zen.hosts_provider: ec2
#discovery.type: ec2
#discovery.zen.ping.multicast.enabled: false
http.port: 9205
#network.host: _eth0_, _local_, _ec2_
network.host: <private ip_addr>
#network.bind_host: <private ip_addr>
#network.publish_host: <private ip_addr>
node.data: true
node.master: true
plugin.mandatory: discovery-ec2, repository-s3
transport.tcp.port: 9305
discovery.zen.ping.unicast.hosts: ["<private ip_addr of node1>:9305","<private ip_addr of node2>:9305"]
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
node.name: nodetest2
#Paths to log, conf, data directories
When both the nodes are started, the following is the log data on both the nodes:
[INFO ][o.e.b.BootstrapCheck ] [nodetest1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[WARN ][o.e.n.Node ] [nodetest1] timed out while waiting for initial discovery state - timeout: 30s
[INFO ][o.e.h.HttpServer ] [nodetest1] publish_address {<private ip_addr of node1>:9205}, bound_addresses {<private ip_addr of node1>:9205}
[INFO ][o.e.n.Node ] [nodetest1] started
[INFO ][o.e.d.z.ZenDiscovery ] [nodetest1] failed to send join request to master [{nodetest}{YcGzQ-4CQtmuuxUGMQJroA}{yuxHmvGPTeK-iw59VTj4ZA}{<private ip_addr of node2>}{<private ip_addr of node2>:9305}{aws_availability_zone=us-east-1d}], reason [RemoteTransportException[[nodetest][<private ip_addr of node2>:9305][internal:discovery/zen/join]]; nested: NotMasterException[Node [{nodetest}{YcGzQ-4CQtmuuxUGMQJroA}{yuxHmvGPTeK-iw59VTj4ZA}{<private ip_addr of node2>}{<private ip_addr of node2>:9305}{aws_availability_zone=us-east-1d}] not master for join request]; ], tried [3] times
I have searched many similar issues and tried to apply the fixes but i still have the same result. Is there any fault in the elasticsearch.yml file?
curl -XGET <private ip_addr>:9205/_cat/master
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
The two node instances are running ES-5.1.1 and are in the same security-group and iam role. Any suggestions are highly appreciated. Thanks,