We are using com.microsoft.azure.kusto:kusto-spark_3.0_2.12:5.0.1 through spark packages:
spark_session = SparkSession.builder \
..
.config('spark.jars.packages', \
"com.microsoft.azure.kusto:kusto-spark_3.0_2.12:5.0.1,"
Then we invoke kusto read as:
df_reader = (
self.spark.read.option("kustoCluster", self.data_source.cluster)
.option("kustoDatabase", self.data_source.database)
.option("kustoQuery", resource.query)
.options(**datasource_options)
)
df = df_reader.load()
Full stacktrace:
: java.lang.NoClassDefFoundError: io/vavr/CheckedFunction0
at com.microsoft.kusto.spark.datasource.DefaultSource.createRelation(DefaultSource.scala:72)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
Caused by: java.lang.ClassNotFoundException: io.vavr.CheckedFunction0
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
We are using spark 3.3.1. What might be happening here?