spring cloud stream kafka DLQ producer config for max.request.size -- more than 1MB failing to produnce message to DLQ - RecordTooLargeException

Question

I'm using the following version in spring cloud streams with kafka, it is a Spring Boot (3.1.5) app:

1. spring-cloud-function - 4.0.5
2. spring-cloud-stream - 4.0.4
3. spring-integration - 6.1.4
4. spring-integration-kafka - 6.1.4

Scenario: I'm producing and consuming a large message upto 15MB from/to a topic successfully and I've a DLQ topic configured to produce the message to DLQ topic if consumer fails to process that message.

Issue: When the producer is failing to process, the spring framework is trying to push the message to the DLQ topic and failing with the following message:

---

"Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 13002359 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration".

In the above case the message size is around 13 MB.

The following is my consumer and DLQ configuration:

  kafka:
    bindings:
      consumeProcess-in-0:
        consumer:
          enable-dlq: true
          dlq-name: processor-error #DLQ topic name
          dlq-producer-properties:
            compression-type: zstd
            configuration:
              max.request.size: 20971520 #20MB
            topic:
              properties:
                max.message.bytes: 20971520  #20MB
          topic: #This is for consumer topic properties
            properties:
              compression.type: zstd
              #max.partition.fetch.bytes: 20971520
              #fetch.message.max.bytes: 20971520  #20MB 
              

I've the following binder config: binder: brokers: ${dekaflow2-kafka-bootstrap-servers} replication-factor: 3 min-partition-count: ${min.topic.partition.count} required-acks: all transaction: transaction-id-prefix: infopost-tx-${random.uuid} configuration: processing.guarantee: exactly_once auto.offset.reset: latest delete.topic.enable: false ...

When try to consume a message which is less than 1MB and fail, I don't see any issue producing the message to DLQ topic. Issue is only when the message size is over 1MB.

Is there anything wrong in my configuration? Is there any other way overriding the default 1MB limit?

Noticed the configuration related to transaction in the binder configuration causing the issue. When I comment out transaction configuration section in the binder configuration it works. If the transaction configuration is on, the DLQ producer config is being defaulted to 1MB otherwise taking it as 20MB. Assuming this as spring cloud streams defect. Any insights over this issue greatly appreciated.