Databricks Certified Associate Developer for Apache Spark free question 3 / 10 · Domain 2: Using Spark SQL
During one run, a PySpark program calls df.createOrReplaceTempView("orders_v") and df.write.saveAsTable("orders_t"). The program is then restarted and connects to the same metastore. According to the Spark SQL data sources guide, what can the restarted program still use?
- orders_v, because createOrReplaceTempView stores the view in the metastore too.
- orders_t, because saveAsTable materializes the data and records it in the metastore.
- Neither, because data saved during a run is removed when that program is restarted.
- Both, because each call writes the data under the warehouse directory of the session.
Show the answer and explanation
Answer: B
Why this is correct
DataFrames can be saved as persistent tables in the Hive metastore with saveAsTable; an existing Hive deployment is not needed, because Spark creates a default local metastore if necessary. Unlike createOrReplaceTempView, saveAsTable materializes the contents of the DataFrame and creates a pointer to the data in the metastore. Persistent tables still exist after the program has restarted, as long as it connects to the same metastore. So orders_t is available again, for example with spark.table("orders_t"), while orders_v belonged to the earlier run.
Why the other options are wrong
"orders_v, because createOrReplaceTempView stores the view in the metastore too." … The guide contrasts the two: unlike the createOrReplaceTempView command, saveAsTable materializes the contents and creates a pointer in the metastore. A temporary view is not the call that records anything there.
"Both, because each call writes the data under the warehouse directory of the session." … Writing data to a table path under the warehouse directory is what saveAsTable does when no custom path is given. createOrReplaceTempView does not write the DataFrame anywhere.
"Neither, because data saved during a run is removed when that program is restarted." … The guide says persistent tables still exist after the Spark program has restarted, as long as the connection to the same metastore is kept, which is the case here.
Getting the table back
The guide notes that a DataFrame for a persistent table can be created by calling the table method on a SparkSession with the name of the table. If the restarted program used a different metastore, the table would not be found, even though its files are still on storage.