Databricks Certified Data Engineer Professional free question 6 / 10 · Domain 5: Cost & Performance Optimization
A notebook reads the same Delta table many times with df = spark.read.table("sales.orders").cache(). After another job updates orders, the notebook sometimes still returns old rows. Which fix follows the Databricks recommendation?
- Keep df.cache() and run REFRESH TABLE sales.orders before each read; changes to a Delta table are picked up by REFRESH TABLE
- Choose a storage level that keeps data on disk with df.persist(); a Spark cache on disk detects file changes and refreshes itself
- Stop using df.cache() and rely on the disk cache; it detects when files are created, deleted, or modified and invalidates stale entries
- Set spark.databricks.io.cache.enabled to false and keep using df.cache(); the disk cache is blocking updates of the Spark cache
Show the answer and explanation
Answer: C
Why this is correct
The Databricks disk cache automatically stores a local copy of Parquet data files when a node fetches them from remote storage. It automatically detects when data files are created, deleted, modified, or overwritten and updates its content accordingly, so stale entries are invalidated and evicted without you doing anything. Databricks recommends using the disk cache, not the Apache Spark cache, for repeated reads of Parquet and Delta Lake data.
Why the other options are wrong
"Keep df.cache() and run REFRESH TABLE sales.orders before each read; changes to a Delta table are picked up by REFRESH TABLE" … This keeps the Apache Spark cache that you create by hand, which the documentation does not describe as detecting file changes. Databricks recommends the disk cache for repeated reads of Delta Lake data, because it detects changed files and invalidates stale entries on its own.
"Set spark.databricks.io.cache.enabled to false and keep using df.cache(); the disk cache is blocking updates of the Spark cache" … The cause of the old rows is the Apache Spark cache you create by hand. The disk cache detects file changes and invalidates stale entries on its own, and turning it off would not refresh the Spark cache and would only slow repeated reads.
"Choose a storage level that keeps data on disk with df.persist(); a Spark cache on disk detects file changes and refreshes itself" … Changing the storage level does not stop it from being an Apache Spark cache that you manage by hand. The cache that detects file changes and refreshes itself is the Databricks disk cache.
Key point
The documentation advises against the Spark cache with Delta Lake because filters added on top of a cached DataFrame lose data skipping, and the cache is not refreshed when the table is read through a different identifier. The disk cache is enabled automatically on certain worker instance types.