Kafka message not deserializable. How to debug

Question

I have kafka topics on which I produce messages in avro format. I use the schema registry. The code is quite basic and looks like this:

// address are changed to localhost.
static {
    props = new Properties();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.KafkaAvroSerializer");
    props.put(ProducerConfig.CLIENT_ID_CONFIG, "Kafka Avro MyObject Producer");
    props.put("schema.registry.url", "http://localhost:8081/");
}

public static void main(String[] args) throws InterruptedException {

    KafkaProducer<String, MyObject> producer = new KafkaProducer<>(props);
    MyObject myObject = createMyObject(); // Create avro object
    final ProducerRecord<String, MyObject> record = new ProducerRecord<>("MyTopic", null, myObject); // null as key

    producer.send(record);
    producer.flush();
    System.out.println("Produced 1 record.");
}

At some point I had a kafka broker issue and had to restart it.

After that for one of my topic (and only one among 30) I started having the error:

org.apache.kafka.common.errors.SerializationException: Error retrieving Avro unknown schema for id 0

I had to go back in the offset of the partitions to determine until which offset it could deserialize object again and noticed that it matched the kafka down time.

This happens on the consumer side. The main one is kafka connect. But in order to test I also used the kafka-avro-console-consumer that gave me the same error.

On the schema registry I also have errors such as:

io.confluent.rest.exceptions.RestNotFoundException: Schema 0 not found

And also the correspondig http request in the schema registry:

"GET /schemas/ids/0?fetchMaxId=false&subject=%3A.%3A HTTP/1.1" 404 51

Firstly, the id 0 is not the one I should be requesting. For this topic the id should be 65 and it does exist in the schema registry.

Secondly the id 0 does not exist in the schema registry.

So another attempt to understand what happend is to check what was the content of an actual message.

I for the most part some unreadable UTF-8 bytes returned but I can recognise some character that indicate me that the message are still written.

For some reason the first octet read is 0, I assume that the consumer expect it to be the schemas id to fetch it and deserialized but since it is 0 it can't fetch anythin.

I'm really confused and hope to receive idea to undertstand what's happening.