Unable to configure GeoSpark in Spark Session :

Question

I have been trying to configure geospark with Spark Session so that I can use spatial application on PySpark. I followed this link & tried to run the code which is mentioned as follows.

try:
     import pyspark
     from pyspark import SparkContext, SparkConf
     from pyspark.sql import SparkSession, SQLContext
except ImportError as e:
     raise ImportError('PySpark is not Configured')

print(f"PySpark Version : {pyspark.__version__}")

# Creating a Spark-Context
sc = SparkContext.getOrCreate(SparkConf().setMaster('local[*]').set("spark.ui.port", "4050"))
# Spark Builder
spark = SparkSession.builder.appName('GeoSparkDemo').config('spark.executor.memory', '5GB')\
    .getOrCreate()

from geospark.register import upload_jars
from geospark.register import GeoSparkRegistrator
upload_jars()
GeoSparkRegistrator.registerAll(spark)

When I run this file it gives me below error.

Traceback (most recent call last): File "c:\sourav\spark\code\geospark_demo.py", line 29, in GeoSparkRegistrator.registerAll(spark) File "C:\Users\user3.conda\envs\python37\lib\site-packages\geospark\register\geo_registrator.py", line 26, in registerAll cls.register(spark) File "C:\Users\user3.conda\envs\python37\lib\site-packages\geospark\register\geo_registrator.py", line 31, in register return spark._jvm.GeoSparkSQLRegistrator.registerAll(spark._jsparkSession) TypeError: 'JavaPackage' object is not callable

I tried to manually add below jar files in spark jars folder

•geospark-1.3.1.jar •geospark-sql_2.1-1.3.1.jar •geo_wrapper.jar

Now the earlier error is gone & new exception is throwing which is as follows:

Traceback (most recent call last): File "c:\sourav\spark\code\geospark_demo.py", line 29, in GeoSparkRegistrator.registerAll(spark) File "C:\Users\user3.conda\envs\python37\lib\site-packages\geospark\register\geo_registrator.py", line 26, in registerAll cls.register(spark) File "C:\Users\user3.conda\envs\python37\lib\site-packages\geospark\register\geo_registrator.py", line 31, in register return spark._jvm.GeoSparkSQLRegistrator.registerAll(spark._jsparkSession)
File "C:\Users\user3.conda\envs\python37\lib\site-packages\py4j\java_gateway.py", line 1257, in call answer, self.gateway_client, self.target_id, self.name) File "C:\sourav\spark\spark-2.4.7-bin-hadoop2.7\python\pyspark\sql\utils.py", line 63, in deco return f(*a, **kw) File "C:\Users\user3.conda\envs\python37\lib\site-packages\py4j\protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.datasyslab.geosparksql.utils.GeoSparkSQLRegistrator.registerAll. : java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.registerFunction(Ljava/lang/String;Lscala/Function1;)V at org.datasyslab.geosparksql.UDF.UdfRegistrator$$anonfun$registerAll$1.apply(UdfRegistrator.scala:29) at org.datasyslab.geosparksql.UDF.UdfRegistrator$$anonfun$registerAll$1.apply(UdfRegistrator.scala:29) at scala.collection.immutable.List.foreach(List.scala:392) at org.datasyslab.geosparksql.UDF.UdfRegistrator$.registerAll(UdfRegistrator.scala:29) at org.datasyslab.geosparksql.utils.GeoSparkSQLRegistrator$.registerAll(GeoSparkSQLRegistrator.scala:34) at org.datasyslab.geosparksql.utils.GeoSparkSQLRegistrator.registerAll(GeoSparkSQLRegistrator.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Unknown Source)

I found this link which had similar issues, I even tried to add jars in spark config files with following code, but nothing seems to worked.

spark.driver.extraClassPath C:\sourav\spark\geosparkjar/*

I'm using Geospark 1.3.1, Java 8, Python 3.7, Apache Spark 2.4.7, my JAVA_HOME, SPARK_HOME is set correctly, I'm running on windows 10.

how can I fix this to proceed further? Any help/suggestion would be highly appreciated.

Answer 1

GeoSpark is currently available as Apache Sedona .

For a similar usecase, I followed the instructions :

pip install apache-sedona

from pyspark.sql import SparkSession
from sedona.utils.adapter import Adapter
from sedona.register import SedonaRegistrator
from sedona.utils import KryoSerializer, SedonaKryoRegistrator
spark = SparkSession.builder.master("spark://test:7077").appName("sedonatest").
    config("spark.serializer", KryoSerializer.getName). \
    config("spark.kryo.registrator", SedonaKryoRegistrator.getName). \
    config('spark.jars.packages',
           'org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.0-incubating,'
           'org.datasyslab:geotools-wrapper:geotools-24.0').getOrCreate()
SedonaRegistrator.registerAll(spark)

resultsDF = spark.sql("SELECT ST_PolygonFromText('-74.0428197,40.6867969,-74.0421975,40.6921336,-74.0508020,40.6912794,-74.0428197,40.6867969', ',') AS polygonshape")

P.S : Pass below 2 Jars during spark submit with --jars option :

• sedona-python-adapter-3.0_2.12-1.0.1-incubating.jar
• geotools-wrapper-geotools-24.0-sources.jar (https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/geotools-24.0/)