Why is difference between sqlContext.read.load and sqlContext.read.text?

Question

I am only trying to read a textfile into a pyspark RDD, and I am noticing huge differences between sqlContext.read.load and sqlContext.read.text.

s3_single_file_inpath='s3a://bucket-name/file_name'

indata = sqlContext.read.load(s3_single_file_inpath, format='com.databricks.spark.csv', header='true', inferSchema='false',sep=',')
indata = sqlContext.read.text(s3_single_file_inpath)

The sqlContext.read.load command above fails with

Py4JJavaError: An error occurred while calling o227.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org

But the second one succeeds?

Now, I am confused by this because all of the resources I see online say to use sqlContext.read.load including this one: https://spark.apache.org/docs/1.6.1/sql-programming-guide.html.

It is not clear to me when to use which of these to use when. Is there a clear distinction between these?

Answer 1

The difference is:

• text is a built-in input format in Spark 1.6
• com.databricks.spark.csv is a third party package in Spark 1.6

To use third party Spark CSV (no longer needed in Spark 2.0) you have to follow the instructions on spark-csv site, for example provide

 --packages com.databricks:spark-csv_2.10:1.5.0  

argument with spark-submit / pyspark commands.

Beyond that sqlContext.read.formatName(...) is a syntactic sugar for sqlContext.read.format("formatName") and sqlContext.read.load(..., format=formatName).

Answer 2

Why is difference between sqlContext.read.load and sqlContext.read.text?

sqlContext.read.load assumes parquet as the data source format while sqlContext.read.text assumes text format.

With sqlContext.read.load you can define the data source format using format parameter.

---

Depending on the version of Spark 1.6 vs 2.x you may or may not load an external Spark package to have support for csv format.

As of Spark 2.0 you no longer have to load spark-csv Spark package since (quoting the official documentation):

NOTE: This functionality has been inlined in Apache Spark 2.x. This package is in maintenance mode and we only accept critical bug fixes.

That would explain why you got confused as you may have been using Spark 1.6.x and have not loaded the Spark package to have csv support.

Now, I am confused by this because all of the resources I see online say to use sqlContext.read.load including this one: https://spark.apache.org/docs/1.6.1/sql-programming-guide.html.

https://spark.apache.org/docs/1.6.1/sql-programming-guide.html is for Spark 1.6.1 when spark-csv Spark package was not part of Spark. It happened in Spark 2.0.

---

It is not clear to me when to use which of these to use when. Is there a clear distinction between these?

There's none actually iff you use Spark 2.x.

If however you use Spark 1.6.x, spark-csv has to be loaded separately using --packages option (as described in Using with Spark shell):

This package can be added to Spark using the --packages command line option. For example, to include it when starting the spark shell

---

As a matter of fact, you can still use com.databricks.spark.csv format explicitly in Spark 2.x as it's recognized internally.