Error: App Search is unable to connect to Elasticsearch. Ensure a healthy Elasticsearch cluster is running at http://localhost:9200 for user elastic

Question

I am trying to connect App Search to the Elastic Search and both of them run in a separate docker containers but the website is not part of the docker.

so, I have a virtual host from the digital ocean, the website is running in /var/www/

My Elastic Search runs on port 9200 in a docker container and My App Search runs on port 3002 in a docker container.

I have a single node (for elastic search).

I get below error when I set the

network.host: "127.0.0.1"

app_search.listen_host: "127.0.0.1" 
elasticsearch.host: http://localhost:9200

Error:

[app-server][INFO]: Failed to connect to Elasticsearch backend. Make sure it is running.

Error: App Search is unable to connect to Elasticsearch. Ensure a healthy Elasticsearch cluster is running at http://localhost:9200 for user elastic.

If I set below in elasticsearch.yml

network.host: 0.0.0.0

and below in app-search.yml

app_search.listen_host: 0.0.0.0
elasticsearch.host: http://elasticsearch:9200

I can get the App search to connect with the Elastic search but then I get below warning from Digital Ocean:

A recent network security scan suggests your Droplet XXXX-web1 is running elasticsearch and that it may be unintentionally exposing data, or misconfigured to allow unauthorized access.

Elasticsearch listens for traffic from everywhere on port 9200 and you can validate this report by attempting to connect to your elasticsearch on 9200 via a simple telnet command:

telnet XXX.XXX.XX.XXX 9200

If the connection is successful you will receive output like the following, which will confirm that your service is visible to the public Internet:

Trying XXX.XXX.XX.XXX...
Connected to XXX.XXX.XX.XXX.
Escape character is '^]'.

You will want to restrict outside access to your Elasticsearch instance to prevent outsiders from reading your data or shutting down your Elasticsearch cluster through its REST API.

Remediation of this issue will take just a few minutes and is relatively straightforward.You will need to open /etc/elasticsearch/elasticsearch.yml and uncomment (remove the "#") and replace its value with 'localhost' so it looks like this:

network.host: localhost

My Operating System is

Ubuntu 18.04

Docker

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:56 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:02 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Elastic Search

elasticsearch:7.6.0

App Search

app-search:7.6.0

Docker Compose file

version: '3.4'

services:
    #MySQL Service
    db:
        image: mysql:5.7.22
        container_name: db
        restart: unless-stopped
        tty: true
        ports:
          - "${HOST_DB_PORT}:3306"
        environment:
          MYSQL_ROOT_PASSWORD: "${DB_ROOT_PASSWORD}"
          MYSQL_DATABASE: "${DB_DATABASE}"
          MYSQL_USER: "${DB_USERNAME}"
          MYSQL_PASSWORD: "${DB_PASSWORD}"
          MYSQL_ROOT_HOST: "%"
        volumes:
            - ./data/dbdata:/var/lib/mysql/:delegated
            - ./docker/mysql/my.cnf:/etc/mysql/my.cnf
        networks:
            - internal


    #Elastic Search Service
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
        container_name: elasticsearch
        restart: unless-stopped
        environment:
            - "node.name=elasticsearch"
            - "discovery.type=single-node"
            - "cluster.name=app-search-docker-cluster"
            - "bootstrap.memory_lock=true"
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        volumes:
           - ./data/elasticsearch:/var/lib/elasticsearch/data/:delegated
           - ./docker/elastic/elasticsearch/elasticsearch.yml:/etc/elasticsearch/elasticsearch.yml
           - ./docker/elastic/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
        ulimits:
          memlock:
            soft: -1
            hard: -1
        ports:
            - 9200:9200
        networks:
            - internal

    #App Search Service
    appsearch:
        image: docker.elastic.co/app-search/app-search:7.6.0
        container_name: appsearch
        restart: unless-stopped
        depends_on:
          - elasticsearch
        environment:
          - "APP_SEARCH_EXTERNAL_URL=${APP_SEARCH_EXTERNAL_URL}"
          - "APP_SEARCH_SECRET_SESSION_KEY=${APP_SEARCH_SECRET_SESSION_KEY}"
          - "ELASTIC_SEARCH_PASSWORD=${ELASTIC_SEARCH_PASSWORD}"
          - "APP_SEARCH_DEFAULT_PASSWORD=${APP_SEARCH_DEFAULT_PASSWORD}"
          - "elasticsearch.host=http://localhost:9200"
          - "JAVA_OPTS=-Xmx1g"
        volumes:
          - ./docker/elastic/appsearch/app-search.yml:/usr/share/app-search/config/app-search.yml
        ports:
          - 3002:3002
        networks:
          - internal

#Docker Networks
networks:
    internal:
        driver: bridge

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: "127.0.0.1"
xpack.security.enabled: true
action.auto_create_index: ".app-search-*-logs-*,-.app-search-*,+*"

app-search.yml

allow_es_settings_modification: true
app_search.listen_host: "127.0.0.1"
elasticsearch.host: http://localhost:9200
filebeat_log_directory: /var/log/app-search
log_directory: /var/log/app-search
app_search.auth.source: standard
elasticsearch.username: elastic
elasticsearch.password: ${ELASTIC_SEARCH_PASSWORD}
app_search.external_url: ${APP_SEARCH_EXTERNAL_URL}
hide_version_info: true
secret_session_key: ${APP_SEARCH_SECRET_SESSION_KEY}
email.account.enabled: true
email.account.smtp.auth: login
email.account.smtp.starttls.enable: ${APP_SEARCH_SMTP_TLS}
email.account.smtp.host: ${APP_SEARCH_SMTP_HOST}
email.account.smtp.port: ${APP_SEARCH_SMTP_PORT}
email.account.smtp.user: ${APP_SEARCH_SMTP_USER}
email.account.smtp.password: ${APP_SEARCH_SMTP_PASSWORD}

Answer 1

I found the solution.

I wanted to prevent public access of the elastic search and the app search to the outside world but accessible by the host server. I did the following:

docker-compose.yml

ports:
    - 127.0.0.1:9200:9200
    - 127.0.0.1:9300:9300
    - "elasticsearch.host=http://elasticsearch:9200"

By adding 127.0.0.1 to the ports makes sure elastic search is accessible to the Main server hosting docker but not the outside world.

The elasticsearch.yml has below

cluster.name: "docker-cluster"
network.host: 0.0.0.0
xpack.security.enabled: true (optional)
action.auto_create_index: ".app-search-*-logs-*,-.app-search-*,+*"

The app-search.yml has below

allow_es_settings_modification: true
app_search.listen_host: 0.0.0.0 (anyone can listen)
elasticsearch.host: http://elasticsearch:9200 (host = elasticsearch)
filebeat_log_directory: /var/log/app-search
log_directory: /var/log/app-search
app_search.auth.source: standard
elasticsearch.username: elastic
elasticsearch.password: ${ELASTIC_SEARCH_PASSWORD}
app_search.external_url: ${APP_SEARCH_EXTERNAL_URL}