Class io.confluent.kafka.serializers.subject.TopicNameStrategy could not be found

93 Views Asked by At

I have a jar that is to be deployed on a custom java based application which requires the application jar and all of its dependencies to be OSGi compliant. I have so far integrated my application jar as well as all of the dependencies shown by maven by manually converting them to OSGi. The jar is required to run as ClassName.MethodName syntax, basically a static class invocation, when I attempt to run my kafka producer program I used to get below error,

Caused by: java.lang.ClassNotFoundException: io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe cannot be found by kafka-avro-serializer-7_0.0.0
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:511)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)

I came across this post on stackoverflow where a user says the issue could be due to AbstractKafkaSchemaSerDe exists in kafka-schema-serializer JAR and its super class AbstractKafkaAvroSerializer exists in kafka-avro-serializer JAR and that first jar loads the package fine where as second one gets skipped due to having same package name. On this line I tried creating a custom jar that is a concatenation of kafka-schema-serializer and kafka-avro-serializer and when I use this custom jar as dependency and load the jar in the classpath I was able to resolve the error ClassNotFound for AbstractKafkaSchemaSerDe.

But now I get a all new error for another Class TopicNameStrategy that is used within the jar as below, it does not say NoClassDef found so I am unsure how to go about debugging this.

org.apache.kafka.common.KafkaException: Failed to construct kafka producer
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:440)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:291)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:274)
    at com.abc.def.InvProducer.publishcommunityProjectEvent(InvProducer.java:36)
    at com.abc.def.PayloadProcessor.publishKafka(PayloadProcessor.java:129)
    at com.abc.def.PayloadProcessor.buildAndSend(PayloadProcessor.java:119)
    at com.abc.def.Kafkaproducer.send_message(Kafkaproducer.java:27)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
    at tool.unknown_source_1018547.execute(Evaluated-inline:1)
    at com.vendor.tool.compiler.serialization.ClassManager.invokeWithoutRecordingClasses(ClassManager.java:127)
    at com.vendor.tool.runtime.ExpressionEvaluator.runLastLoadedClass(ExpressionEvaluator.java:673)
    at com.vendor.tool.compiler.serialization.CompiledToolLoader.loadRealToolClass(CompiledToolLoader.java:61)
    at com.vendor.tool.compiler.serialization.BLETransient.load(BLETransient.java:46)
    at com.vendor.tool.runtime.ExpressionEvaluator.runEntries(ExpressionEvaluator.java:629)
    at com.vendor.tool.runtime.ExpressionEvaluator.runTreeRecordingResults(ExpressionEvaluator.java:612)
    at com.vendor.tool.runtime.ExpressionEvaluator.compileAndRunParseTree(ExpressionEvaluator.java:511)
    at com.vendor.tool.runtime.ExpressionEvaluator.compileAndRunParseTree(ExpressionEvaluator.java:402)
    at com.vendor.tool.runtime.ExpressionEvaluator.compileAndRunForValueWithReporter(ExpressionEvaluator.java:297)
    at com.vendor.tool.runtime.ExpressionEvaluator.compileAndRunForValue(ExpressionEvaluator.java:353)
    at com.vendor.tool.tools.REPL.processInputLines(REPL.java:123)
    at com.vendor.tool.tools.REPL.processInputLinesFromConsole(REPL.java:143)
    at tool.sw_kernel.sys_src.tool_rep_4273.tool_rep__process_command(sys_src/misc/tool_rep.tool:136)
    at tool.sw_kernel.sys_src.tool_rep_4273.tool_rep__cli(sys_src/misc/tool_rep.tool:90)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
    at com.vendor.tool.language.invokers.OneShotInvokeHelper.invokeInternal(OneShotInvokeHelper.java:39)
    at com.vendor.tool.language.invokers.OneShotInvokeHelper.invoke(OneShotInvokeHelper.java:21)
    at com.vendor.tool.runtime.ToolRoot.invoke(ToolRoot.java:131)
    at com.vendor.tool.tools.REPL.bombproofLoop(REPL.java:176)
    at tool.sw_kernel.sys_src.system_6601.system__session_start(sys_src/guts/system.tool:3225)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
    at com.vendor.tool.language.invokers.OneShotInvokeHelper.invokeInternal(OneShotInvokeHelper.java:39)
    at com.vendor.tool.language.invokers.OneShotInvokeHelper.invoke(OneShotInvokeHelper.java:21)
    at com.vendor.tool.runtime.ToolRoot.invoke(ToolRoot.java:131)
    at com.vendor.tool.tools.REPL.handleConsole(REPL.java:100)
    at com.vendor.tool.tools.ConsoleActivator$ConsoleStarter.run(ConsoleActivator.java:71)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ExceptionInInitializerError
    at io.confluent.kafka.serializers.KafkaAvroSerializer.configure(KafkaAvroSerializer.java:50)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:368)
    ... 35 more
Caused by: org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.kafka.serializers.subject.TopicNameStrategy for configuration key.subject.name.strategy: Class io.confluent.kafka.serializers.subject.TopicNameStrategy could not be found.
    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744)
    at org.apache.kafka.common.config.ConfigDef$ConfigKey.<init>(ConfigDef.java:1146)
    at org.apache.kafka.common.config.ConfigDef.define(ConfigDef.java:152)
    at org.apache.kafka.common.config.ConfigDef.define(ConfigDef.java:172)
    at org.apache.kafka.common.config.ConfigDef.define(ConfigDef.java:211)
    at org.apache.kafka.common.config.ConfigDef.define(ConfigDef.java:373)
    at org.apache.kafka.common.config.ConfigDef.define(ConfigDef.java:386)
    at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig.baseConfigDef(AbstractKafkaSchemaSerDeConfig.java:161)
    at io.confluent.kafka.serializers.KafkaAvroSerializerConfig.<clinit>(KafkaAvroSerializerConfig.java:43)
    ... 37 more
org.apache.kafka.common.KafkaException: Failed to construct kafka producer

When I run this program locally within Eclipse it works fine so I suspect it is due to specific order of class loading or issue with OSGI manifest. If I use any other serializer such as Custom Byte serializer the program works fine, the issue is only with KafkaAvroSerializer. I believe the issue is specific to AbstractKafkaSchemaSerDeConfig class when the baseConfigDef method gets loaded it is unable to load or find TopicNameStrategy class.

 public static final String KEY_SUBJECT_NAME_STRATEGY_DEFAULT =
      TopicNameStrategy.class.getName();

I do not know what else I could try to fix this. I am using below version of schema and avro serializer libraries. Much appreciated if anyone can point me in the right direction.

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>6.2.2</version>
</dependency>
<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-schema-serializer</artifactId>
    <version>6.2.2</version>
</dependency>
0

There are 0 best solutions below