Elasticsearch Low watermark setting triggered by disk partitions

Question

My index is in 'yellow' state because one shard is unassigned. My three data nodes have plenty of disk space overall but they are partitioned. Each data node has three disk partitions called /data1, /data2 and /data3. Elasticsearch is only storing shards on /data1. Following this page from the manual, I modified elasticsearch.yml on the head node to include the following:

path.data: /data1/elasticsearch, /data2/elasticsearch_2, /data3/elasticsearch_3

And then ran:

curl -XPOST "localhost:9200/_cluster/reroute?retry_failed"

To try and re-allocate the shards. However, I am getting the warning the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85% from to of the nodes and the shard cannot be allocated to the same node on which a copy of the shard already exists from the third data node.

There is a second, much smaller index also on this cluster, which in green status.

My question is, how can I get Elasticsearch to store shards on the additional partitions of the data nodes when the first partition is full? Each partition by itself is large enough to store one shard but I think because I have data for multiple indices in one partition it is triggering the warning and preventing the distribution of the replica shards.

Answer 1

indices cannot relocated when cluster is yellow. I recommend to set number_of_replica to 0 and set cluster.routing.allocation.disk.watermark.low and cluster.routing.allocation.disk.watermark.high to 95%. then wait for indices relocated. after that set these setting to defaults (1, 85%, 95%).

note that cluster.routing.allocation.total_shards_per_node default is -1 and check its value.