I wanted to upgrade the spring-boot version of my project from 2.4.4 to 2.6.7. Because of this, the kafka-client version also increased from 2.6.0 to 3.0.1. There were few other version upgrades but the only relevant one for the issue I am about to describe was kafka-reactor upgrade from 1.2.11.RELEASE to 1.3.11.
We load tested the change on our pre-production environment and it worked fine there. However, after release we started getting the following exception and had to rollback our release:
org.apache.kafka.common.errors.UnsupportedForMessageFormatException: The message format version on the broker does not support the request.
Upon investigation we figured out that the only difference bewteen our pre-prod and prod Kafka setup was the message format version, log.message.format.version. In pre-prod it was 2.0 but in prod it was 0.10.0.
What I don't understand is why a new Kakfa-client version won't work with an old broker version? Should it not be backward-compatible?
Is there any way I can fix it without actually changing the broker config?
0.10.0 is ancient (if that's your broker version).
I am not sure, but perhaps with the new major version (3.0.x), they dropped support for really old brokers.
In any case, spring-kafka 2.8.5, which comes with Boot 2.6.7, also works with the 2.8.1 kafka-clients, so you could try downgrading to that version.
https://docs.spring.io/spring-kafka/docs/2.8.1/reference/html/#update-deps
But you really need to look at moving your prod broker to a much newer version.