Packetbeat not able to connect to elasticsearch docker

Question

I am trying to dockerize all the elastic services that I need to use. The docker-compose file looks like below

version: '3'
services:
  redis:
    build: ./docker/redis

  postgresql:
    build: ./docker/postgresql
    ports:
      - "5433:5432"
    env_file:
      - .env

  graphql:
    build: .
    command: npm run start
    volumes:
      - ./logs/:/usr/app/logs/
    ports:
      - "3000:3000"
    env_file:
      - .env
    depends_on:
      - "redis"
      - "postgresql"
    links:
      - "redis"
      - "postgresql"

  elasticsearch:
    build: ./docker/elasticsearch
    container_name: elasticsearch
    networks:
      - elastic
    ports:
      - "9200:9200"
    depends_on:
      - "graphql"
    links:
      - "kibana"

  kibana:
    build: ./docker/kibana
    container_name: kibana
    ports:
      - "5601:5601"
    depends_on:
      - "graphql"
    networks:
      - elastic
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200

  metricbeat:
    build: ./docker/metricbeat
    depends_on:
      - "graphql"
      - "elasticsearch"
      - "kibana"
    volumes:
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
    networks:
      - elastic
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    command:
      - "-system.hostfs=/hostfs"

  packetbeat:
    build: ./docker/packetbeat
    depends_on:
      - "graphql"
      - "elasticsearch"
      - "kibana"
    cap_add:
      - NET_ADMIN
    networks:
      - elastic
    environment:
      - ELASTICSEARCH_URL=http://127.0.0.1:9200

  logstash:
    build: ./docker/logstash
    ports:
      - "9600:9600"
    volumes:
      - ./logs:/usr/logs
    depends_on:
      - "graphql"
      - "elasticsearch"
      - "kibana"
    networks:
      - elastic
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200

networks:
  elastic:
    driver: bridge

Everything works very well right now but the problem is that the packetbeat is only capturing network inside its own docker container. In the elastic documentation reference - https://www.elastic.co/guide/en/beats/packetbeat/master/running-on-docker.html It says that I need to enable 'host' network in order to capture all the originating and arriving networks to the physical host. However, since I have configured the networks to be -elastic I am unable to add additional host network interface to packetbeat. If I erase -elastic network and add -host network, I am not able to connect to elasticsearch because DNS elasticsearch no longer exists in a different network. How can I overcome this problem?

Answer 1

This is a pretty common problem where the nice isolation of docker gets in your way. The same happens for example when using the Prometheus node_exporter that collects metrics of the host machine, which is also pretty useless when run in a container without access to the host network.

As you already mentioned, it is not possible to use both network_mode: host and the docker networks togehter. So for your use case, you could have the packetbeat container running with host networking and not attach it to the docker networks. Because of that, you are no longer able to connect it to the elasticsearch instance via http://elasticsearch:9200, so you need to replace this config value to http://your-host-ip:9200 which you already configured in your elasticsearch service as mapped port. Possibly http://127.0.0.1 could also work when run with network_mode: host as this should be the localhost in your host network - thus the host where the port of elasticsearch is mapped to.