kafka kraft cluster (3.5.0) with sasl/scram unable to start

Question

i want to delete zookeeper from my kafka cluster architecture and use the kraft mode, but i am struggling to make it work. it works like a charm with PLAINTEXT, SSL and SASL/PLAIN mechanism but not with sasl/scram-**.

the point is that i want to use multiple users which is not possible with SASL/PLAIN (only with jaas config statis file) hence the scram mechanism.

what is strange is that it works with only one node controller.quorum.voters=101@host-101:9093 the cluster starts and i can add users with kafka-configs.`

when i add an other node

controller.quorum.voters=101@host-101:9093,102@host-102:9093

the cluster does not start and i have an authentication failed even if i've done everything as expected by the documentation:

STEPS:

• declare a log dir and format it with a cluster-id and --add-scram in all nodes before running up the cluster.

./bin/kafka-storage format --config etc/kafka/kraft/server_sasl_scram_plain.properties --cluster-id CLUSTER_ID--add-scram 'SCRAM-SHA-256=[name=admin,password=admin]'

CONFIG:

process.roles=broker,controller

node.id=101

listeners=SASL_PLAINTEXT://:9092,CONTROLLER://:9093
security.inter.broker.protocol=SASL_PLAINTEXT
controller.listener.names=CONTROLLER

listener.security.protocol.map=CONTROLLER:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL



#SASL
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.controller.protocol=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256

listener.name.sasl_plaintext.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin"
 password="admin";

listener.name.controller.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admi
n" password="admin";

ERROR:

[2023-10-10 16:09:04,308] ERROR [kafka-101-raft-outbound-request-thread]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=kafka.raft.KafkaNetworkChannel$$Lambda$777/0x0000000840540440@523f2af4, destination=102, correlationId=72, clientId=raft-client-101, createdTimeMs=1696946943990, requestBuilder=VoteRequestData(clusterId='OqaN99lZTN2EsWJJL9Zbzw', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=55, candidateId=101, lastOffsetEpoch=2, lastOffset=1206)])])) (kafka.raft.RaftSendThread)
[2023-10-10 16:09:04,308] ERROR Request OutboundRequest(correlationId=72, data=VoteRequestData(clusterId='OqaN99lZTN2EsWJJL9Zbzw', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=55, candidateId=101, lastOffsetEpoch=2, lastOffset=1206)])]), createdTimeMs=1696946943990, destinationId=102) failed due to authentication error (kafka.raft.KafkaNetworkChannel)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256

The credentials are OK, the ports are open but still have authentication failed.. any idea please? i wonder if ever someone has made it with the sasl/scram

Note : i have tested with apache kafka 3.5.0 and confluent plateform (CE) 7.5.0