Databricks Certified Associate Developer for Apache Spark free question 1 / 10 · Domain 1: Apache Spark Architecture and Components
A developer reads the driver's logs and sees the terms job, stage and task. Using the glossary in the Spark cluster mode overview, which description puts these units in the right relationship?
- A job is divided into stages that depend on each other, and a task is sent to one executor.
- A task is split into stages, and each stage is sent to one executor as a single unit of work.
- A stage contains several jobs, and each job is a set of tasks run in parallel on one executor.
- A job is the unit sent to one executor, and its stages are spread over the worker nodes.
Show the answer and explanation
Answer: A
Why this is correct
The glossary of the cluster mode overview gives the three levels. A job is a parallel computation consisting of multiple tasks. Each job gets divided into smaller sets of tasks called stages that depend on each other, similar to the map and reduce stages in MapReduce. A task is a unit of work that will be sent to one executor. So an application runs jobs, each job is split into dependent stages, and each stage is made of tasks that executors run. The glossary also notes that job and stage are the terms you will see used in the driver's logs.
Why the other options are wrong
"A stage contains several jobs, and each job is a set of tasks run in parallel on one executor." … The glossary has the containment the other way round: each job gets divided into smaller sets of tasks called stages. A stage is part of a job, not a container of jobs, and the tasks of a stage are not tied to a single executor.
"A task is split into stages, and each stage is sent to one executor as a single unit of work." … The glossary defines the task as the smallest unit: a unit of work that will be sent to one executor. Stages are sets of tasks, so a task is not split into stages, and a stage is not shipped to an executor as one piece.
"A job is the unit sent to one executor, and its stages are spread over the worker nodes." … A job is a parallel computation consisting of multiple tasks, so it is not the unit that goes to one executor. That role belongs to the task, and the job's work is spread out task by task.
Reading the hierarchy in the logs and the UI
When a stage waits on another, it is because stages depend on each other inside a job; when a stage is slow, look at its tasks, since tasks are what executors actually run. The application sits above all of this: it consists of a driver program and executors on the cluster, and it can run many jobs over its lifetime.