We are using kafka 0.10.2.0 and trying to use HDFS Sink connector to push the data from Kafka to HDFS. This succeeds only for the first time and after that no update is happening. Also, the Kafka standalone connector log has:
[2017-09-01 19:20:03,030] INFO WorkerSinkTask{id=hdfs-sinks-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:272)
[2017-09-01 19:20:03,034] WARN Commit of WorkerSinkTask{id=hdfs-sinks-0} offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask:172)
Below are the properties added as suggested in one of the forums.
name=hdfs-sinks
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=mysql-author
hdfs.url=hdfs://quickstart.cloudera:8020
flush.size=3
logs.dir=/hdfs_directory/data/log
request.timeout.ms=310000
offset.flush.interval.ms=5000
heartbeat.interval.ms=60000
session.timeout.ms=300000
max.poll.records=100
Connector Config:
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=120000
offset.flush.timeout.ms=60000
buffer.memory=100