In Apache Spark, the execution hierarchy begins when an action is called on a DataFrame or RDD, which triggers a job. Spark's scheduler analyzes the Directed Acyclic Graph (DAG) of transformations and breaks the job into one or more stages. Stages are sets of computations that can be executed together without a data shuffle. Each stage is then composed of multiple tasks, which are the smallest units of execution. Each task runs on a single executor core and processes a single partition of data. This hierarchical model is Job -> Stage -> Task.

A. This is incorrect because stages contain tasks, not the other way around. A task is the most granular unit of execution.

B. This is incorrect because jobs contain stages. A stage is a subdivision of a job, not a container for multiple jobs.

C. This is incorrect as it inverts the entire hierarchy. A job is the highest-level unit, and a task is the lowest.

1. Apache Spark Official Documentation - Glossary:

Job: "A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save

collect)..."

Stage: "Each job gets divided into smaller sets of tasks called stages that depend on each other..."

Task: "A unit of work that will be sent to one executor."

Source: Apache Spark 3.4.1 Documentation

"Glossary".

2. Databricks Documentation - Spark UI - Jobs Tab:

"The Jobs tab displays a summary of all jobs in the Spark application... The job detail page shows a visualization of the DAG. In the DAG

vertices represent the RDDs or DataFrames and the edges represent the operations to be applied... The DAG is also organized into stages." This documentation visually and textually confirms that jobs are broken into stages

which in turn consist of tasks.

Source: Databricks Documentation

"Spark UI - Jobs tab".

3. Learning Spark

2nd Edition (by Databricks employees):

Chapter 13

"How Spark Executes a Program

" page 304: "When the driver runs

it converts the user’s program into units of physical execution called tasks. Each task is a combination of a chunk of data and a computation to be performed on that chunk. All of this is orchestrated by the driver

which launches tasks on the cluster. A set of tasks is called a stage

and a set of stages is called a job."