Having an issue wherein the load on ES nodes in terms of load factor, CPU usage or indexing rate is not uniform and some nodes are having higher load than others. This is causing poor performance (high latency) on writes/reads even as overall utilization is <60%. Need some suggestions on how to improve load balancing.
Large cluster: 60 nodes, heap size 16 GB, RAM 52 GB, dual core RHEL. Ingestion: 200k docs/second, cluster size 5-6 TB. Some indexes are much larger than others and have higher number of configured shards. 1 replica per shard. Indexes are rolled over daily (new index created daily). Read rate is 10% of write rate. refresh interval 60s.
If I further increase #shards per index to allow the load to be more uniformly balanced across the cluster, how does it affect read/write path? Are there other recommeneded approaches to manage load balancing and improve cluster performance?