Spring Cloud Stream Kinesis Binder 4.0 doesn't support Multi Stream with KCL 2 Enabled

Question

KCL 2 allows multi stream support with the same consumer group in the same application.

With cloud stream kinesis binder 4.0, kpl-kcl-enabled: true A single stream in the destination works.

bindings:
   kinesis-read-binding:
      destination: stream1
      group: consumergroupname
      binder: kinesis

The lease table defines the leaseKey as shardId.

When trying to read multiple streams on a single binding with the consumer group as shown below, the application runs in faulty mode by overriding the leases for the other stream shards.

bindings:
   kinesis-read-binding:
      destination: streamA,streamB
      group: consumergroupname
      binder: kinesis

I have looked in the source code to look at the possible error, below is the cause that I think is the problem.

When not in multiplex mode, the BindingService iterates over each stream name in bindConsumer function and creates a different instance of KCLMessageDrivenChannelAdapter from the bindConsumer. Due to the same, the leaseKey defined in the table doesn't use the MultiStreamTracker hence overrides the shard for other streams.

With multiplex mode, KCLMessageDrivenChannelAdapter should be called with multiple streams, but fails much before due to

bindings:
   kinesis-read-binding:
      destination: streamA,streamB
      group: consumergroupname
      binder: kinesis
      consumer:
         multiplex: true    

KinesisStreamProvisioner. The KinesisStreamProvisioner.provisionConsumerDestination() fails and throws exception.

// In KinesisStreamProvisioner.java the below statement throws Error from the createOrUpdate function
// name = "streamA,streamB"
return new KinesisConsumerDestination(name, createOrUpdate(name, shardCound));

private List<Shard> createOrUpdate(String stream, int shards) {
try{
// the below statement fails. stream = "streamA,streamB"
  shardList = getShardList(stream).join()
   ..
   ..
}

The above doesn't allow spring cloud stream kinesis binder to run with MultiStream Mode supported by KCL 2.x.

Answer 1

This is the bug in Spring Cloud Stream Kinesis Binder: we just miss to handle that multiplex option in the logic you are mentioning.

Please, raise a GH issue and we will address it as soon as possible.