Kafka command line producer/consumer have 1 second latency

Question

I'm doing a test run with Kafka using the command line producer and consumer.

I'm running this in one Terminal window

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tag7

and this in another

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tag7 --zookeeper localhost:2181

but it takes over 1 second for the data that I sent to be printed by the consumer. The data that I'm sending is what I type in to the producer, so basically 1 message every few seconds. Are there any configuration options I can change so that the Kafka broker expects very few messages per second and thus makes the messages move significantly quicker?

I'm using the default configurations of Zookeeper and Kafka, so I haven't configured much.

Thank you in advance!

Answer 1

There are two configuration parameters for Kafka - one is setting the minimum amount of data received before answering consumer request and the other is setting the maximum amount of time to wait for this data to arrive before answering the request.

You can try to add following options:

--consumer-property fetch.max.wait.ms=0 --consumer-property fetch.min.bytes=0

For more info:

• Look at this issue for details.
• Check the kafka consumer configs.

Answer 2

As the answer from Ivan Georgiev didn't work for me (not only because it is about the consumer and not the producer), I opened another question here in StackOverFlow, and finally also in Apache's Jira and ppatierno (thanks) answered why it happened and how to solve it.

For changing the console producer's properties linger.ms (1000ms by default) and batch.size (16384 by default) you need to specify it in the command line with --timeout and --max-partition-memory-bytes respectively, for example, if you are in the Kafka's main folder:

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testConsole --timeout 100