kcat protobuf deserialization

Question

I'm using kcat to check the content of kafka topics when working locally but, when messages are serialized with protobuf, the result I get is an unreadable stream of encoded characters. I'm aware of the existence of some other kafka-consumers tools (Kafdrop, AKHQ, Kowl, Kadek...) but I'm looking for the simplest option which fits my needs.

Does kcat support protobuf key/value deserialization from protofile?
Is there any simple terminal-based tool which allows this?

Answer 1

I've had luck with this command:

kcat -C -t <topic> -b <kafkahost>:9092 -o -1 -e -q -D "" | protoc --decode=<full message class> path/to/my.proto --proto_path <proto_parent_folder>

Update 12/2023 - Here's a more streamlined command with better guidance.

kcat -C -t <topic> -b <kafkahost>:9092 -o -1 -e -q -D "" | protoc --decode <package>.<Message Type> path/to/my.proto

Where is the package defined in the .proto file and <Message Type> is the name of the Message.

Ie.: user.proto:

package some.package;

message User {
  string id = 1;
  string email = 2;
}

The command would be:

kcat -C -t <topic_name> -b host.docker.internal:9092 -o -1 -e -q -D "" | protoc --decode some.package.User user.proto

Answer 2

any simple terminal-based tool which allows this

Only ones that integrate with the Confluent Schema Registry (which is what those linked tools use as well), e.g. kafka-protobuf-console-consumer is already part of Confluent Platform.

Regarding kcat - refer https://github.com/edenhill/kcat/issues/72 and linked issues

Answer 3

any simple terminal-based tool which allows this

I created milena for exactly this use case. It's a small CLI for working with protobuf Kafka messages that decodes protobuf to json for ease of use with other tools such as jq.

For example, the following command would consume protobuf some.package.User events and write them to stdout as line delimited json.

milena consume \
  -X bootstrap.servers=host.docker.internal:9092 \
  --topic my-topic \
  --file-descriptors descriptors.binpb \
  --message-name some.package.User \
  --offset=beginning