We are using solr 8.9.0 version and we are facing replica recovery loop issue occasionally.
Problem:
we do have zoo keeper instances having two shards in two different servers having cross server replica's for each shard.
Now at some time over production we facing issue like replica gone in recovery and than its getting down and due to that we acing issue in whole platform for searching other functionality.
We have checked it some slow queries and possible ways find using googling but didn't find any proper way for the same. (zookeeper logs, slow query logs, other solr configuration..and many more things.)
Is there any way to check specific thing to identify the same ?
What is the best way to do load balancing as here we do have one more shard but at the time of replica recovery its not able to serve other requests ?
Is it good to use Apache curator/solrctl or any third party framework for fix service discovery issue ?
Any other suggestions ?
Thanks in advance !