I have working configuration for confluentinc/cp-kafka that works fine in docker-compose as following:
User
kafka-kraft:
image: confluentinc/cp-kafka:7.5.1
container_name: kafka-kraft
hostname: kafka-kraft
ports:
- "9101:9101"
- "9092:9092"
environment:
- KAFKA_NODE_ID=1
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-kraft:29092,PLAINTEXT_HOST://localhost:9092
- KAFKA_JMX_PORT=9101
- KAFKA_JMX_HOSTNAME=localhost
- KAFKA_PROCESS_ROLES=broker,controller
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-kraft:29093
- KAFKA_LISTENERS=PLAINTEXT://kafka-kraft:29092,CONTROLLER://kafka-kraft:29093,PLAINTEXT_HOST://0.0.0.0:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
- KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
Now I try to configure it so it will work with SASL_PLAINTEXT authentication, tried to change it to the following configuration:
kafka-kraft:
image: confluentinc/cp-kafka:7.5.1
container_name: kafka-kraft
hostname: kafka-kraft
ports:
- "9101:9101"
- "9092:9092"
environment:
- KAFKA_NODE_ID=1
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT_HOST:SASL_PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://kafka-kraft:29092,PLAINTEXT_HOST://localhost:9092
- KAFKA_JMX_PORT=9101
- KAFKA_JMX_HOSTNAME=localhost
- KAFKA_PROCESS_ROLES=broker,controller
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-kraft:29093
- KAFKA_LISTENERS=SASL_PLAINTEXT://kafka-kraft:29092,CONTROLLER://kafka-kraft:29093,PLAINTEXT_HOST://0.0.0.0:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=SASL_PLAINTEXT
- KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
- KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
- KAFKA_SASL_ENABLED_MECHANISMS=PLAIN
- KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
volumes:
- ./kafka_server_jaas.conf:/etc/kafka/kafka_server_jaas.conf
kafka_server_jaas.conf:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_kafka="kafka-secret";
};
Now when I try to run it, I get the following error:
===> User
2023-11-29T10:52:36.043745169Z uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2023-11-29T10:52:36.043961086Z ===> Configuring ...
2023-11-29T10:52:36.047186169Z Running in KRaft mode...
2023-11-29T10:52:36.625520503Z CLUSTER_ID is required.
2023-11-29T10:52:36.625564128Z Command [/usr/local/bin/dub ensure CLUSTER_ID] FAILED !
Honestly, I first time try to configure Kafka for developing environment and have no clue what I need to fix to cause it to work, and why adding security have an impact on CLUSTER_ID at all?