Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArraySerializer

Question

i am doing spark structured streaming with kafka and cassandra while i am running below command i got an error

spark-submit --class StreamHandler --master local[*] --packages "org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.3" target/scala-2.12/stream-handler_2.12-1.0.jar

I am doing this project Ubuntu linux machine itself

causedby:java.lang.ClassNotFoundException:org.apache.kafka.common.serialization.ByteArraySerializerat java.net.URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 24 more

How to resolve the error?

My versions:

• spark - spark-3.2.3-hadoop2.7
• scala - 2.12.15
• kafka - kafka_2.12-2.8.0
• sbt - sbt-1.9.1

Answer 1

You should create an uber JAR with SBT that includes kafka-clients and spark-sql-kafka-0-10, rather than use --packages. Otherwise, you'll need to provide both.