How to run KSQLDB along with Kafka server using docker compose

Question

I'm trying to run KSQLDB along with Kafka, but KSQLDB refuses to connect to my Kafka Server.

kafka1-kafka-1   | [2024-02-19 17:08:15,202] INFO [BrokerLifecycleManager id=1] Unable to register broker 1 because the controller returned error DUPLICATE_BROKER_REGISTRATION (kafka.server.BrokerLifecycleManager)
kafka1-ksqldb-1  | [2024-02-19 17:08:16,320] INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka1-ksqldb-1  | [2024-02-19 17:08:16,321] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.18.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka1-kafka-1   | [2024-02-19 17:08:16,809] INFO [QuorumController id=1] registerBroker: failed with DuplicateBrokerRegistrationException in 293 us. Reason: Another broker is registered with that broker id. (org.apache.kafka.controller.QuorumController)
kafka1-kafka-1   | [2024-02-19 17:08:16,813] INFO [BrokerLifecycleManager id=1] Unable to register broker 1 because the controller returned error DUPLICATE_BROKER_REGISTRATION (kafka.server.BrokerLifecycleManager)

My docker-compose.yaml looks like this:

version: "3.1"
services:
  kafka:
    image: confluentinc/confluent-local:latest
    ports:
      - "9092:9092"
      - "8082:8082"
      - "29093:29093"
  ksqldb:
    image: confluentinc/cp-ksqldb-server:7.5.3
    environment:
      - KSQL_BOOTSTRAP_SERVERS=kafka:9092
      - KSQL_KSQL_SERVICE_ID=ksqldb_
      - KSQL_KSQL_QUERIES_FILE=/opt/data/queries.sql
      - KSQL_LISTENERS=http://0.0.0.0:8088/
    depends_on:
      - kafka

What I need is basically a docker-compose file that can run the two images and let me experiment with tables and streams.

Answer 1

You're missing some environment variables on the Kafka container

KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092" 
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"

It defaults to localhost, so no other containers can connect

And you need KSQL_BOOTSTRAP_SERVERS=kafka:29092 on ksql (change the port)