How Akka stream internal and explicit buffer interact with the underlying kafka client settings in alpakka Kafka?

158 Views Asked by At

I am trying to use Akka stream buffer to improve the throughput of my stream, I'm wondering how it does apply to Kafka

Consumer.committableSource(consumerSettings, Subscriptions.topics(topic))

in particular,

 val kafkaSource =
   Consumer.committableSource(consumerSettings, Subscriptions.topics(topic))
     .buffer(10000, OverflowStrategy.backpressure)

What exactly happen here with respect to the underlying Kafka API?

I have the following configuration on the underlying Kafka client:

.withProperty(AUTO_OFFSET_RESET_CONFIG, offsetReset)
      .withProperty(MAX_POLL_INTERVAL_MS_CONFIG, maxPollIntervalMs.toString)
      .withProperty(SESSION_TIMEOUT_MS_CONFIG, sessionTimeoutMs.toString)
      .withProperty(HEARTBEAT_INTERVAL_MS_CONFIG, heartbeatIntervalMs.toString)
      .withProperty(FETCH_MAX_WAIT_MS_CONFIG, fetchMaxWaitMs.toString)
      .withProperty(MAX_POLL_RECORDS_CONFIG, maxPollRecords.toString)
      .withProperty(FETCH_MAX_BYTES_CONFIG, maxPollRecords.toString)
      .withProperty(MAX_PARTITION_FETCH_BYTES_CONFIG, maxPollRecords.toString)

Hence I have a MAX_POLL_RECORDS_CONFIG, FETCH_MAX_BYTES_CONFIG and MAX_PARTITION_FETCH_BYTES_CONFIG

What I wonder is how the buffer would play with respect to the fetching configured on the underlying client.

Is Consumer.committableSource materialized in its own Actor, and receive message from the underlying Kafka client, through its buffer? Let's say the underlying client is configured to fetch up to a million messages, and the Actor as a buffer of 1000? What does that mean? What happens?

Does the Actor buffer override the poll request of the Kafka client, or does it gets data in its mail box which the Kafka client push to, until the result of its poll (the max configured in the underlying client) has been passed through?

I think I most need to know how the internal and or explicit buffer of Kafka stream interacts with the settings of the poll request.

0

There are 0 best solutions below