I'm trying to create a cluster of three servers with dotCMS 5.2.6 installed. They have to interface with a second cluster of 3 elasticsearch nodes. Despite my attempts to combine them, the best case I've obtained is with both dotCMS and elastic up and running but from dot admin backend (Control panel > Configuration > Network) I always see my three servers with red status due to Index red status.
I have tested the following combinations:
In plugins/com.dotcms.config/conf/dotcms-config-cluster-ext.properties
AUTOWIRE_CLUSTER_TRANSPORT=false
es.path.home=WEB-INF/elasticsearch
Using AUTOWIRE_CLUSTER_TRANSPORT=true seems not to change the result
In plugins/com.dotcms.config/ROOT/dotserver/tomcat-8.5.32/webapps/ROOT/WEB-INF/elasticsearch/config/elasticsearch-override.yml
transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: first_es_server:9300, second_es_server:9300, third_es_server:9300
Using transport.tcp.port: 9300 cause dotCMS startup failure with error:
ERROR cluster.ClusterFactory - Unable to rewire cluster:Failed to bind to [9300]
Caused by: com.dotmarketing.exception.DotRuntimeException: Failed to bind to [9300]
Of course, port 9300 is listening on the three elasticsearch nodes they are configured with transport.tcp.port: 9300 and have no problem to start and create their cluster.
Using transport.tcp.port: 9301 dotCMS can start and join the elastic cluster but the index status is always red even if the indexation seems to work and nothing is apparently affected.
Using transport.tcp.port: 9309 (as suggested in the dotCMS online reference) or any other port number lead to the same result as 9301 case but from dot admin backend (Control panel > Configuration > Network) the Index information for each machine still repot 9301 as ES port.
Main Question
I would like to know where the ES port can be edited considering my Elasticsearch cluster is performing well (all indices are green) and the elasticsearch-override.yml within dotCMS plugin doesn't affect the default 9301 reported by the backend.
Is the HTTP interface enabled on ES? If not, I would enable it and see what the cluster health is and what the index health is. It might be that you need to adjust your expected replicas.
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-health.html
and
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html
FWIW, the upcoming version of dotCMS (5.3.0) does not support embedded elasticsearch and requires a vanilla external ES node/custer to connect to.