I am migrating the existing spark streaming application from spark2.3 to spark3.1.1. I have updated below-mentioned spark dependencies
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>3.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.1.1</version>
</dependency>
And have added phoenix dependencies
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-core</artifactId>
<version>4.13.2-cdh5.11.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-spark</artifactId>
<version>4.13.2-cdh5.11.2</version>
<scope>provided</scope>
</dependency>
While running spark job I am getting java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
. The same code used to run with spark2.3 but now its not with spark3.1.1.
java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
Can someone please help me to know what am I missing here?