Write Data in JSON to KAFKA Using Avro and Confluent schema registry using NiFi

Question

I would like to write data in JSON format to a Kafka Cluster using Avro and a Confluent schema registry. I already created a schema in the confluent schema registry, which looks like this:

The JSON looks like this:

In NiFi I'M currently using the PublishKafkaRecord_2_6 processor which is configured like this:

To process the JSON I'm using a JSONTreeReader which is configured like this:

To Write to Kafka we are Using AvroRecordSetWriter which is configured like this:

When I have a look what is written into Kafka I get something cryptic like this:

Can somebody maybe point out my mistake?

Thanks in advance.

Answer 1

Avro is a binary format that happens to have a JSON representation. The AvroRecordSetWriter uses the binary format only. To write JSON out, you'd need to use the JSON record set writer.

So all you're seeing, as far as I can tell, is the binary representation of your messages in the Kafka queues. That would be the expected behavior using the Avro writer with the processor.

FWIW, this is precisely the pattern you should be using with Kafka in most costs. The binary format is smaller and more concise, so Kafka will process it on both ends faster. You can also set it up to not write the schema into the binary blob, so it'll be really concise and then just have downstream systems use a convention based on topic name or something to pick a schema.