(SEE UPDATE BELOW)
I am trying to read events from a kafka topic using Apache Druid (v29.0.0). I successfully connected to the Kafka clusters and schema registry, so I am able to read the serialized data along with fetching the schemas.
When I go to deserialize the data read from the kafka topic I am getting the error Failed to sample data: com.google.protobuf.Descriptors$DescriptorValidationException: <schema subject field> ".google.protobuf.Timestamp" is not defined. I also noticed that the classes in druid/lib/protobuf-java.jar all exist in druid/extensions/druid-protobuf-extensions/druid-protobuf-extensions.jar but I am not sure if the issue is a dependency/classpath issue or something else.
Any help is greatly appreciated.
UDPATE: RESOLVED
For context, I was following this tutorial https://druid.apache.org/docs/latest/development/extensions-core/protobuf/#when-using-schema-registry to enable consuming protobuf messages from kafka.
In my environment, common types (beginning with google.protobuf.*) were purposely left out of a schemas references. Since kafka-protobuf-provider-6.0.1.jar does not have the functionality to automatically add the common types if they are not present in the schema reference, druid was not able to resolve the types (see error above). Had the google.protobuf.* types been included in my schema's references, I would not have encountered this error.
Instead you need to use version kafka-protobuf-provider-6.2.0.jar or higher (see this commit in confluent schema-registry for context). And to make this work, I needed to import additional jars (all of them from https://mvnrepository.com/artifact/io.confluent). These are the jars I now have under extensions/druid-protobuf-extensions:
druid-protobuf-extensions % ls -ltr
- druid-protobuf-extensions-29.0.0.jar
- kotlin-stdlib-1.4.0.jar
- kafka-protobuf-provider-6.2.0.jar
- kafka-protobuf-types-6.2.0.jar
- wire-runtime-3.6.0.jar
- wire-schema-3.6.0.jar
- proto-google-common-protos-2.36.0.jar
Hope this helps anybody encountering the same issue!