kafka connect to s3 fails to start

Question

I'm trying to configure the kafka-connect to send my data from kafka to s3. I'm newbie in aspect of kafka, and I'm trying to implement this flow without any ssl encryptions just to get the hang of it.

kafka version : 2.12-2.2.0
kafka-connect : 4.1.1 (https://api.hub.confluent.io/api/plugins/confluentinc/kafka-connect-s3/versions/4.1.1/archive)

In the server.properties file the only change that I did is setting the advertised.listeners to my ec2 IP:

advertised.listeners=PLAINTEXT://ip:9092

kafka-connect properties :

# Kafka broker IP addresses to connect to
bootstrap.servers=localhost:9092

# Path to directory containing the connector jar and dependencies
plugin.path=/root/kafka_2.12-2.2.0/plugins/

# Converters to use to convert keys and values
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

# The internal converters Kafka Connect uses for storing offset and configuration data
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
security.protocol=SASL_PLAINTEXT
consumer.security.protocol=SASL_PLAINTEXT

my s3-sink.properties file :

name=s3.sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
topics=my_topic
s3.region=us-east-1
s3.bucket.name=my_bucket
s3.part.size=5242880
flush.size=3
storage.class=io.confluent.connect.s3.storage.S3Storage
format.class=io.confluent.connect.s3.format.json.JsonFormat
schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
schema.compatibility=NONE

I'm launcing kafka-connect with the following command :

connect-standalone.sh kafka-connect.properties s3-sink.properties

At first I got the following error :

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set

From other posts I saw that I need to create a jaas config file so that what I did :

cat config/kafka_server_jass.conf
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="userName"
serviceName="kafka"
password="password";
};

and :

export KAFKA_OPTS="-Djava.security.auth.login.config=/root/kafka_2.12-2.2.0/config/kafka_server_jass.conf"

Now I'm getting the following error :

Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login

help :)

Answer 1

You might also need to define principal and keytab inside your jaas configuration:

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="userName" 
  serviceName="kafka"
  password="password";
  useKeyTab=true
  keyTab="/etc/security/keytabs/kafka_server.keytab"
  principal="kafka/[email protected]";
};