Export a spark logical/physical plan?

Question

Can one export a Spark logical or physical plan of a dataframe/set, serialize it and save it somewhere (as text, xml, json ...). Then re-import it, and create a dataframe based on it ?

The idea here is, I'm interested in having a metastore for Spark dataframes where I can save dataframes logical or physical plans, so that others could use them.

Answer 1

spark 2.4.2 below code may be different for lower version of spark.

Check below code.

spark.read.json(Seq(df.queryExecution.logical.toJSON).toDS).write.format("json").save("logical")
spark.read.json(Seq(df.queryExecution.sparkPlan.toJSON).toDS).write.format("json").save("sparkPlan")
spark.read.json(Seq(df.queryExecution.executedPlan.toJSON).toDS).write.format("json").save("executedPlan")
spark.read.json(Seq(df.queryExecution.analyzed.toJSON).toDS).write.format("json").save("analyzed")