Kafka UI not able to connect to Broker

Question

I am new to Containerisation. I am trying to set up my local env where my java application want to connect to Kafka. Can't use Docker so decided to use Podman. I have three containers running on same network each for Kafka, Zookeeper and kafka UI. I am using macos.

Here are the Podman commands i am using :-

Creating network:

podman network create kafka-network --driver bridge

Spinning up Zookeper: podman run --name zookeeper-server -p 2181:2181 --network kafka-network -e ALLOW_ANONYMOUS_LOGIN=yes docker.io/bitnami/zookeeper:3.8

spinning up Kafka: podman run --name kafka-server --network kafka-network -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:1 -p 9092:9092 docker.io/bitnami/kafka:3.4

Spinning up Kafka-ui: podman run -it --name kafka-ui -p 9090:8080 --network kafka-network -e KAFKA_CLUSTERS_0_NAME=local -e KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper-server:2181 -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-server:9092 -e DYNAMIC_CONFIG_ENABLED=true docker.io/provectuslabs/kafka-ui:latest

Url used by java Application: PLAINTEXT://localhost:9092

Java Application is able to connect as partitions are getting assigned but kafka ui log says "Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available."

If I change localhost to kafka-server in podman command where I am spinning up kafka then Kafka UI is working but consumers are not subscribing to partitions in java App.

Not sure why I am seeing this strange behaviour. Extra explanation is welcome as I am trying to understand the purpose of environment variables here. Thanks

Answer 1

Although it's Docker, I had this same experience when wanting to try out kafka-ui, and couldn't get it to work until this morning. Thanks to trying out a different UI, some config there hinted that broker:9092 should also be provided as a bootstrap server.

To clarify, I'm using Docker Compose with 3 services, zookeeper, broker and kafka-ui, as in below:

---
version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.3.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TOOLS_LOG4J_LOGLEVEL: ERROR

  kafka-ui:
    image: provectuslabs/kafka-ui
    hostname: kafka-ui
    container_name: kafka_ui
    depends_on:
    - broker
    ports:
    - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
      KAFKA_CLUSTERS_0_NAME: kafkacluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker:29092,localhost:29092,localhost:9092,broker:9092'

In the above config, I previously had broker:29092 but not broker:9092. After adding broker:9092, it became alive! (^^,)

To verify that it is indeed the solution, I reverted to a config without broker:9092 and again I had the same issue as before.

I hope this helps!