Does REFRESH TABLE update the cache entries of all tables?

410 Views Asked by At

I am looking for an approach to update the all the table metadata cache entry just before the write the operation. I have found the way via spark.catalog.refreshTable(table), however I am not sure whether it will update all the tables metadata store which was used in spark.sql() to make the table.

Code Sample:

 val dataFrame1 :DataFramew = ...
 dataFrame1.createOrReplaceTempView("table1")

 val dataFrame2 :DataFramew = ...
 dataFrame2.createOrReplaceTempView("table2")

 val dataFrame3 =  spark.sql(select * from table1 inner join table2 .....)
 dataFrame3.createOrReplaceTempView("table3")

 spark.catalog.refreshTable(table3) 
\\ does the above line of code will update the cached table metadata for table1 and table 2 as well before the write operation? Or do we need to refresh the table separately ?

 dataFrame3.write.parquet("/temp/.....) 
0

There are 0 best solutions below