what will be the PySpark version compatibility for jdk@11, python 3.9, pydeequ 1.1.1 for mac OS?

Question

In my current project, I'm continuously getting "JAVA package not callable" error.

spark session created. connected to postgresql db using jdbc driver. able to load the complete data

but when I'm adding deequ analyzer then getting "java package" load issue.

tried versions using :- java@11, [email protected], [email protected]/[email protected], [email protected]

May I know which version should I use for all these? Note: I'm using PyCharm IDE .

Answer 1

Now, It is working properly.

During spark session creation I was importing pydeeque package and JDBC Driver package into spark.jars.package object, so pydeequ was overrided by jdbc, hence "JAVA package not callable" error was coming.

INCORRECT CODE :-

spark = SparkSession.builder \
    .appName("AppName") \
    .config("spark.jars.packages", pydeequ.deequ_maven_coord) \
    .config("spark.jars.excludes", pydeequ.f2j_maven_coord) \
    .config("spark.jars.packages", "org.postgresql:postgresql:42.6.0") \
    .getOrCreate()

then I tried adding two packages into spark.jar

.config("spark.jars.packages", 'path/package1, path/package2') but it didn't worked.

Lastly I added postgresql jars path to the spark.jars, and now it is working perfectly.

CORRECT CODE :-

    `spark = SparkSession.builder \
    .appName("AppName") \
    .config("spark.jars.packages", pydeequ.deequ_maven_coord) \
    .config("spark.jars.excludes", pydeequ.f2j_maven_coord) \
    .config("spark.jars", "https://jdbc.postgresql.org/download/postgresql-42.6.0.jar") \
    .getOrCreate()`