I am looking for an approach to update the all the table metadata cache entry just before the write the operation. I have found the way via spark.catalog.refreshTable(table)
, however I am not sure whether it will update all the tables metadata store which was used in spark.sql() to make the table.
Code Sample:
val dataFrame1 :DataFramew = ...
dataFrame1.createOrReplaceTempView("table1")
val dataFrame2 :DataFramew = ...
dataFrame2.createOrReplaceTempView("table2")
val dataFrame3 = spark.sql(select * from table1 inner join table2 .....)
dataFrame3.createOrReplaceTempView("table3")
spark.catalog.refreshTable(table3)
\\ does the above line of code will update the cached table metadata for table1 and table 2 as well before the write operation? Or do we need to refresh the table separately ?
dataFrame3.write.parquet("/temp/.....)