Databricks Certified Associate Developer for Apache Spark free question 6 / 10 · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications
A team wants Spark SQL to stop choosing broadcast joins on its own based on table size, for a whole session. Which setting does the Spark SQL performance tuning guide describe for this?
- spark.sql.broadcastTimeout set to 0, so no broadcast has time to finish.
- spark.sql.autoBroadcastJoinThreshold set to its largest possible value.
- spark.sql.autoBroadcastJoinThreshold set to -1, which disables it.
- spark.sql.autoBroadcastJoinThreshold set to 0, the size of no table.
Show the answer and explanation
Answer: C
Why this is correct
The tuning guide lists spark.sql.autoBroadcastJoinThreshold under Automatically Broadcasting Joins. It configures the maximum size in bytes for a table that will be broadcast to all worker nodes when performing a join. The guide adds that by setting this value to -1, broadcasting can be disabled. Setting it to -1 for the session stops Spark SQL from choosing broadcast joins on its own because a table looks small.
Why the other options are wrong
"spark.sql.autoBroadcastJoinThreshold set to 0, the size of no table." … The guide names one specific value for turning automatic broadcasting off, and it is not 0. The threshold is a size in bytes, and the documented way to disable it is a negative value.
"spark.sql.broadcastTimeout set to 0, so no broadcast has time to finish." … spark.sql.broadcastTimeout is the timeout in seconds for the broadcast wait time in broadcast joins. Setting it to 0 does not turn automatic broadcasting off; it changes how long a broadcast may take.
"spark.sql.autoBroadcastJoinThreshold set to its largest possible value." … The threshold is the maximum size in bytes for a table that will be broadcast. A very large value would let more tables be broadcast, which is the opposite of the goal.
Size-based choice and its limits
The threshold is how Spark decides by itself that a table is small enough to send to every worker node. The decision rests on size estimates, so the setting matters most when estimates are wrong. Setting it back to a positive size restores the automatic choice.