Databricks Certified Associate Developer for Apache Spark free question 2 / 10 · Domain 1: Apache Spark Architecture and Components
A cluster has 8 worker nodes running executors with 5 cores each, 40 CPU cores in total. A developer wonders why the cluster looks underused and asks how many tasks a stage should have. What does the Spark tuning guide recommend in general?
- About 40 tasks, one per CPU core, so no core ever waits for a second task.
- About 8 tasks, one per worker node, so each node handles one large piece of work.
- About 80 to 120 tasks, two to three per CPU core across the whole cluster.
- About 400 to 800 tasks, ten to twenty per core, to keep scheduling overhead low.
Show the answer and explanation
Answer: C
Why this is correct
The tuning guide warns that clusters will not be fully utilized unless you set the level of parallelism for each operation high enough. Spark sets the number of map tasks for a file from its size, and for distributed reduce operations it uses the largest parent RDD's number of partitions; you can pass a level of parallelism as an argument or set spark.default.parallelism. As a general rule, the guide recommends 2-3 tasks per CPU core in your cluster. With 8 nodes of 5 cores, that is 40 cores and therefore about 80 to 120 tasks per stage.
Why the other options are wrong
"About 8 tasks, one per worker node, so each node handles one large piece of work." … The guide sizes parallelism by CPU cores, not by nodes. With one task per node, most of the 40 cores would sit idle, which is exactly the under-use the guide warns about when the level of parallelism is not high enough.
"About 40 tasks, one per CPU core, so no core ever waits for a second task." … One task per core is below the guide's figure. The recommendation is 2-3 tasks per CPU core in the cluster, so that cores that finish early can pick up more work instead of waiting for the slowest task.
"About 400 to 800 tasks, ten to twenty per core, to keep scheduling overhead low." … The guide's general figure is 2-3 tasks per CPU core. For the 40 cores here that is 80 to 120 tasks per stage, far below ten to twenty tasks for every core.
Cores, tasks and partitions
A task works on one partition, so the number of tasks in a stage follows the number of partitions. When the Spark UI shows a stage with far fewer tasks than the cluster has cores, the partition count is the lever. Count cores across all executors, not nodes, because each core runs one task at a time by default.