Databricks Certified Data Engineer Associate free question 4 / 10 · Domain 3: Data Transformation and Modeling
On Databricks Runtime 17.3 LTS, a query aggregates with Structured Streaming (stateful processing) and writes to a Delta table, and it has been running with the default value of spark.sql.shuffle.partitions (200). Because the data is small, you want the aggregation shuffle to run with 64 partitions. Which procedure makes it run with 64 partitions?
- Set spark.sql.shuffle.partitions to 64 and resume from the same checkpoint location as before
- Set spark.sql.adaptive.coalescePartitions.minPartitionNum to 64 and resume from the same checkpoint location as before
- Set spark.sql.shuffle.partitions to 64 and start with a new checkpoint location
- Set spark.sql.adaptive.coalescePartitions.minPartitionNum to 64 and start with a new checkpoint location
Show the answer and explanation
Answer: C
Why this is correct
In a stateful streaming query, the number of shuffle partitions is fixed when the checkpoint is created. The documentation says that changing spark.sql.shuffle.partitions has no effect on a query that already has a checkpoint, and that to apply a new number you must start the query with a new checkpoint location. Setting it to 64 and starting with a new checkpoint location makes it run with 64.
Why the other options are wrong
"Set spark.sql.shuffle.partitions to 64 and resume from the same checkpoint location as before" … The note in the documentation configuration table says: "For Structured Streaming, this configuration cannot be changed between query restarts from the same checkpoint location." If you resume from the same checkpoint location, it keeps running with the 200 that was set when the checkpoint was created, even if you set 64.
"Set spark.sql.adaptive.coalescePartitions.minPartitionNum to 64 and start with a new checkpoint location" … coalescePartitions.minPartitionNum is the minimum number of partitions that adaptive query execution (AQE) coalesces to after a shuffle. The documentation describes the queries AQE applies to as non-streaming. Coalescing does not work for a streaming query, so even with a new checkpoint it runs with the 200 of spark.sql.shuffle.partitions.
"Set spark.sql.adaptive.coalescePartitions.minPartitionNum to 64 and resume from the same checkpoint location as before" … minPartitionNum is an AQE coalescing setting, and AQE applies to non-streaming queries. It has no effect on a streaming query, and a query resumed from the same checkpoint keeps running with 200.
Common trap
When you start with a new checkpoint location, the query is treated as new, and neither its earlier progress nor its state is carried over. Decide where to read from again before you switch.