Reading a Kusto source from Spark raises java.lang.ClassNotFoundException: io.vavr.CheckedFunction0

77 Views Asked by At

We are using com.microsoft.azure.kusto:kusto-spark_3.0_2.12:5.0.1 through spark packages:

    spark_session = SparkSession.builder \
        ..
        .config('spark.jars.packages', \
                "com.microsoft.azure.kusto:kusto-spark_3.0_2.12:5.0.1,"

Then we invoke kusto read as:

        df_reader = (
            self.spark.read.option("kustoCluster", self.data_source.cluster)
            .option("kustoDatabase", self.data_source.database)
            .option("kustoQuery", resource.query)
            .options(**datasource_options)
        )

        df = df_reader.load()

Full stacktrace:

: java.lang.NoClassDefFoundError: io/vavr/CheckedFunction0
    at com.microsoft.kusto.spark.datasource.DefaultSource.createRelation(DefaultSource.scala:72)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
Caused by: java.lang.ClassNotFoundException: io.vavr.CheckedFunction0
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)

We are using spark 3.3.1. What might be happening here?

0

There are 0 best solutions below