Kafka Broker Failed Authentication - invalid credentials

Question

For the last day or so I have trying to setup locally using confluent docker images, Kafka cluster with one node. Unfortunately, haven't been able to do so. Below are all my config files:

/etc/kafka/secrets/zookeeper_server_jaas.conf

Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user_admin="admin_secret";
};

/etc/kafka/secrets/kafka_server_jaas.conf

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="admin_secret";
};

Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="admin"
   password="admin_secret";
};

docker-compose.yml

version: '3.5'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/zookeeper_server_jaas.conf
          -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
          -Dzookeeper.allowSaslFailedClients=false
          -Dzookeeper.requireClientAuthScheme=sasl
    volumes:
      - ./secrets:/etc/kafka/secrets
  
  broker:
    image: confluentinc/cp-kafka:latest
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENERS: SASL_SSL://:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL_SSL:SASL_SSL
      KAFKA_ADVERTISED_LISTENERS: SASL_SSL://broker:9092
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512
      KAFKA_INTER_BROKER_LISTENER_NAME: SASL_SSL
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: broker_sslkey_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_truststore_creds
      KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/kafka_server_jaas.conf
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: false
      KAFKA_SSL_CLIENT_AUTH: "required"
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: "HTTPS"
    volumes:
      - ./secrets:/etc/kafka/secrets

Certificates which I am referencing in the docker compose are under /secrets directory on my local machine next docker-compose file itself.

The error I am getting while running docker-compose up is:

broker     | [2023-06-23 09:22:15,816] INFO [Controller id=1, targetBrokerId=1] Failed authentication with broker/192.168.16.3 (channelId=1) (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) (org.apache.kafka.common.network.Selector)
broker     | [2023-06-23 09:22:15,818] INFO [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
broker     | [2023-06-23 09:22:15,818] ERROR [Controller id=1, targetBrokerId=1] Connection to node 1 (broker/192.168.16.3:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (org.apache.kafka.clients.NetworkClient)

Answer 1

Although I haven't tried this personally, can you update the user_admin field to have value as admin in /etc/kafka/secrets/zookeeper_server_jaas.conf. Both kafka and zookeeper should have same user details, as far as I know. I might be wrong.