why is my kafka consumer not showing any messages after i setup SASL_PLAINTEXT

Question

so i installed kafka server and confirmed that they work when i just plainly test them

then i have to set up SASL_PLAINTEXT for the kafka so i followed the instruction from here

which basically created

1. zookeeper_jaas.conf

2. kafka_jaas.conf

then add configuration to

1. zookeeper.properties

zookeeper.sasl.client=true
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl

2. server.properties

super.users=User:admin
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
allow.everyone.if.no.acl.found=true
listeners=SASL_PLAINTEXT://my_ip:9092
advertised.listeners=SASL_PLAINTEXT://my_ip:9092

and then adding the line export KAFKA_OPTS="-Djava.security.auth.login.config=file:$base_dir/../config/zookeeper_jaas.conf" and "kafka_jaas.conf" accordingly in their ...-start.sh

then after starting zookeeper and kafka, i tried

but the consumer shows nothing

this is the consumer.properties

What am i missing?

kafka: version 3.5.0 zookeeper: version 3.6.4 ubuntu: Linux 5.15.0-82-generic

Answer 1

I'm not exactly sure what I did, but I think I just reinstall things with different approach and it works

This is what I do:

1. Install

curl "https://archive.apache.org/dist/kafka/2.1.0/kafka_2.12-2.1.0.tgz" -o ~/Downloads/kafka2.tgz
mkdir kafka2
cd kafka2
tar -xvzf ~/Downloads/kafka2.tgz --strip 1

2. zookeeper service Configuration (zookeeper.service)

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

3. kafka service Configuration (kafka.service)

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka2/bin/kafka-server-start.sh /home/kafka/kafka2/config/server.properties'
ExecStop=/home/kafka/kafka2/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

and then I just start and stop it as a service

sudo service zookeeper start
sudo service kafka start
sudo service kafka status
sudo service kafka stop

My testing Configuration

My Kafka is in my vm and I need to connect to it from outside the vm

Configuration File: kafka2/config/server.properties

listeners={auth mechanism}://0.0.0.0:9092
advertised.listeners={auth mechanism}://myvm:9092
...
security.inter.broker.protocol={auth mechanism}

options (auth mechanism)

• SASL_SSL (SASL + TLS/SSL)
• SASL_PLAINTEXT (SASL only)
• SSL (TLS/SSL only)
• PLAINTEXT

Additional Configuration

1. Use TLS/SSL

ssl.truststore.location=/home/kafka/ssl/kafka.broker0.truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/home/kafka/ssl/kafka.broker0.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.enabled.protocol=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS

2. SASL

sasl.enabled.mechanisms={sasl mechanism}
sasl.mechanism.inter.broker.protocol={sasl mechanism}

options (sasl mechanism)

• PLAIN

# add to config file
listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="admin" \
   password="admin007" \
   user_admin="admin007";

• SCRAM-SHA-256

./kafka-configs.sh --alter --add-config 'SCRAM-SHA-256=[password=admin007],SCRAM-SHA-512=[password=admin007]' --entity-type users --entity-name admin --zookeeper localhost:2181

# add to config file
listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
   username="admin" \
   password="admin007";

• SCRAM-SHA-512

./kafka-configs.sh --alter --add-config 'SCRAM-SHA-256=[password=admin007],SCRAM-SHA-512=[password=admin007]' --entity-type users --entity-name admin --zookeeper localhost:2181

# add to config file
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
   username="admin" \
   password="admin007";