could not be established. Broker may not be available for kafka UI and broker

Question

I am trying to create a kafka cluster using bitnami kafka image and kafka UI with following file. When I use the network mode host, kafka UI can access kafka-broker and everything works however when I use bridge network the UI is not getting connected to the broker.

version: "3"
services:
  kafka-ui:
    container_name: kafka-ui
    image: docker.io/provectuslabs/kafka-ui:latest
    ports:
      - 8080:8080
    depends_on:
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9094
      DYNAMIC_CONFIG_ENABLED: 'true'
    networks:
      - app-tier

  kafka:
    container_name: kafka-single-broker
    image: 'docker.io/bitnami/kafka:latest'
    ports:
      - '9094:9094'
    environment:
      KAFKA_CFG_NODE_ID: 0
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka:9093
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_ENABLE_KRAFT: 'yes'  # Enable KRaft mode
    networks:
      - app-tier

networks:
  app-tier:
    driver: bridge

I know the fix requires some sort of change in lister configuration. Thanks for the help. Also I am using KRAFT instead of zookeeper

Error:
024-03-03 18:59:37,708 WARN  [kafka-admin-client-thread | kafka-ui-admin-1709492351-18] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1709492351-18] Connection to node -1 (kafka/127.0.1.1:9094) could not be established. Broker may not be available.
2024-03-03 18:59:38,711 INFO  [kafka-admin-client-thread | kafka-ui-admin-1709492351-18] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1709492351-18] Node -1 disconnected.
2024-03-03 18:59:38,711 WARN  [kafka-admin-client-thread | kafka-ui-admin-1709492351-18] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1709492351-18] Connection to node -1 (kafka/127.0.1.1:9094) could not be established. Broker may not be available.
2024-03-03 18:59:39,714 INFO  [kafka-admin-client-thread | kafka-ui-admin-1709492351-18] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1709492351-18] Node -1 disconnected.
2024-03-03 18:59:39,714 WARN  [kafka-admin-client-thread | kafka-ui-admin-1709492351-18] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1709492351-18] Connection to node -1 (kafka/127.0.1.1:9094) could not be established. Broker may not be available.

I ran the podman-compose and two containers are getting created. I should have been able to see the online cluster on kafka-ui but the cluster is showing offline. New docker-compose

Answer 1

You have a misconfiguration. In kafka you define listeners for external traffic to listen on localhost only.

KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094

Use kafka service name here and it will work:

KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://kafka:9094