Avro Schema deserialization Index out of bound exception error - schema resolution

Question

I have 2 applications, one is producing the data to topic and another application receiving the data from topic. writer schema version different from reader(consumer) schema. Producer having new schema and where as consumer have old schema. With schema registry schema evolve options I thought consumer works successfully but I am getting below error. Looks like consumer looking ordering of received fields in specificrecord and not ignoring the unknown field.

From Avro docs:

if both are records:

1. The ordering of fields may be different: fields are matched by name.
2. Schemas for fields with the same name in both records are resolved recursively.
3. if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored.

 Caused by: java.lang.IndexOutOfBoundsException: Invalid index: 2
    at com.test.Employee.put(Employee.java:111) ~[classes!/:0.0.1-SNAPSHOT]
    at org.apache.avro.generic.GenericData.setField(GenericData.java:816) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:139) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[avro-1.9.2.jar!/:1.9.2]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[avro-1.9.2.jar!/:1.9.2]

Producer Schema:

        {
    "type": "record",
    "namespace": "com.aa.opshub.test",
    "name": "Employee",
    "fields": [
        {
            "name": "firstName",
            "type": "string"
        },
        {
            "name": "age",
            "type": "int"
        },
        {
            "name": "middleName",
            "type": "string"
        }
    ]
}

Consumer Schema:

  {
   "type" : "record",
    "namespace" : "com.aa.opshub.test",
    "name" : "Employee",
    "fields" : [
        { "name" : "firstName" , "type" : "string" },
        { "name" : "age", "type" : "int" }
    ]
}

Consumer: SpecificRecord class method throwing error

 @SuppressWarnings(value="unchecked")
  public void put(int field$, java.lang.Object value$) {
    switch (field$) {
    case 0: firstName = (java.lang.CharSequence)value$; break;
    case 1: age = (java.lang.Integer)value$; break;
    default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
    }
  }

Answer 1

I was running into the same issue with a client and server using avro for communication. In my case, the fix was to use server-side schema while reading records sent from the client which was apparently on a different schema than my server. I had to change the way I was initialising the reader. I kept the following code instead,

 DatumReader<GenericRecord> reader;
            if(serverSchema != null) {
                reader = new GenericDatumReader<>(serverSchema);
            } else {
                reader = new GenericDatumReader<>();
                LOGGER.warn("Schema version mismatch can cause execution failures. Always pass expected server schema.");
            }