How can one use spark Catalyst?

Question

According to this Spark Catalyst is An implementation-agnostic framework for manipulating trees of relational operators and expressions. I want to use Spark Catalyst to parse SQL DMLs and DDLs to write and generate custom Scala code for. However, it is not clear for me by reading the code if there is any wrapper class around Catalyst that I can use? The ideal wrapper would receive a sql statement and produces the equivalent Scala code. For my use case would look like this

def generate("select substring(s, 1, 3) as from t1") = 
{ // custom code 
 return custom_scala_code_which is executable given s as List[String]
}

This is a simple example, but the idea is that I don't want to write another parser and I need to parse many SQL functionality from a legacy system that I have to write a custom Scala implementation for them.

In a more general question, with a lack of class level design documentation, how can someone learn the code base and make contributions?

Answer 1

Dataframe automatically changes into RDD and optimise the code, and this optimization is done through Catalyst. When a programmer writes a code in Dataframe , internally code will be optimized. For more detail visit

Catalyst optimisation in Spark

Answer 2

Spark takes SQL queries using spark.sql. For example: you can just feed the string SELECT * FROM table as an argument to such as spark.sql("SELECT * FROM table") after having defined your dataframe as "table". To define your dataframe as "table" for use in SQL queries create a temporary view using

DataFrame.createOrReplaceTempView("table")

You can see examples here:

https://spark.apache.org/docs/2.1.0/sql-programming-guide.html#running-sql-queries-programmatically