How to create emptyRDD using SparkSession - (since hivecontext got deprecated)

Question

IN Spark version 1.*

Created emptyRDD like below:

var baseDF = hiveContextVar.createDataFrame(sc.emptyRDD[Row], baseSchema)

While migrating to Spark 2.0(since hiveContext got deprecated, using sparkSession)

Tried like:

var baseDF = sparkSession.createDataFrame(sc.emptyRDD[Row], baseSchema)

Though getting below error:

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243)

Is there a way to create emptyRDD using sparkSession?

Answer 1

In Spark 2.0 you need to refer the spark context through spark session. You can create empty dataframe as below. It worked for me.

sparkSession.createDataFrame(sparkSession.sparkContext.emptyRDD[Row], baseSchema)

Hope it helps you.