Databricks Certified Associate Developer for Apache Spark free question 7 / 10 · Domain 4: Troubleshooting and Tuning Apache Spark DataFrame API Applications
A sort-merge join has a few tasks that run far longer than the rest, because some join keys carry most of the rows. With spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled both on, what does AQE do about it?
- It merges each skewed partition with small neighbours to even out the sizes.
- It splits skewed tasks, replicating if needed, into roughly even-sized tasks.
- It converts the join into a broadcast hash join, for the skewed keys only.
- It raises the number of shuffle partitions for the stage and runs it again.
Show the answer and explanation
Answer: B
Why this is correct
The guide says data skew can severely downgrade the performance of join queries. The skew join feature dynamically handles skew in sort-merge join by splitting, and replicating if needed, skewed tasks into roughly evenly sized tasks. It takes effect when both spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled are enabled. The few long tasks are cut into pieces of similar size, so they no longer hold up the stage.
Why the other options are wrong
"It converts the join into a broadcast hash join, for the skewed keys only." … The guide's skew handling acts on the skewed tasks of the sort-merge join: it splits them, replicating if needed. It does not pick out keys to be joined with a different strategy.
"It merges each skewed partition with small neighbours to even out the sizes." … Merging contiguous partitions is coalescing, which the guide describes as a way to avoid too many small tasks. Merging a large partition with its neighbours would make it larger, not smaller.
"It raises the number of shuffle partitions for the stage and runs it again." … The guide's skew handling acts on the skewed tasks themselves, splitting them, and replicating if needed, into roughly evenly sized tasks. Changing the partition count of the whole stage and running it again is not what the feature does.
Even out the long tasks
If skew handling would introduce an extra shuffle, the guide lists spark.sql.adaptive.forceOptimizeSkewedJoin to force it on anyway, to avoid straggler tasks. It is false by default.