How to perfectly use KPL/KCL with spring-cloud-stream-binder-aws-kinesis lib?

Question

I've decided to use spring-cloud-stream-binder-aws-kinesis lib on version 4.0.1, but I have some questions about how I can implement KPL/KCL with it. I'm currently following this guide on how to utilize the properties: https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/blob/main/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc

From what I tested, if I set kplKclEnabled to true, the enhanced fan-out feature is also enabled by default. Is that statement true? If so, is there a way to avoid using the enhanced fan-out but keep utilizing the KPL/KCL?

With the kplKclEnabled, I also tried to reduce the throughput over my KCL - because I was getting some errors over my applications for I was receiving too much data - by changing the consumerBackoff and recordsLimit default values. Still, it doesn't seem to be affecting the consumer. Does anyone know of a way for me to reduce the throughput? Is it something that I can manage? More importantly, and if I am not asking too much, do you know more details about these settings along with idleBetweenPolls?

Answer 1

Starting with that 4.0.1 version there is option for the consumer:

fanOut:: The KCL retrieval mode: true - fan-out, false - polling.

The recordsLimit and idleBetweenPolls are not propagated to the KclMessageDrivenChannelAdapter.

Feel free to raise a GH issue for Spring Integration AWS to expose those options for the KclMessageDrivenChannelAdapter. However they still will be used only for polling mode.