Apache Spark operates on the principle of lazy evaluation. It builds a Directed Acyclic Graph (DAG) of transformations (read, filter, select, groupBy) but does not execute any computation. The processing is only triggered when an action is called. In the provided code, count() is the first action invoked on the DataFrame reduceddf. This call forces Spark to execute the entire chain of preceding transformations—from reading the CSV file to filtering, selecting, and grouping—to compute the final result, which is the number of rows in the aggregated DataFrame.

A. filter is a narrow transformation. It only adds a step to the logical execution plan and does not trigger any data processing.

C. groupBy is a wide transformation that defines a shuffle operation. However, like all transformations, it is lazy and does not execute until an action is called.

D. show() is also an action that would trigger execution. However, it appears after the count() action in the code. Therefore, the computation begins when count() is called first.

1. Databricks Documentation - Apache Spark programming with Databricks: In the "Transformations" section

it states

"Transformations are lazy. Code in a notebook cell that defines a DataFrame and transformations does not run until you explicitly call an action." It lists count() and show() as common actions. This confirms that processing starts with the first action

count().

Source: Databricks Documentation

Introduction to structured data in PySpark

Section: "Transformations".

2. Apache Spark 3.4.1 Documentation - RDD Programming Guide: The fundamental concept of lazy evaluation is explained here. "All transformations in Spark are lazy

in that they do not compute their results right away... The transformations are only computed when an action requires a result to be returned to the driver program." The guide lists count() as a primary example of an action.

Source: Apache Spark Official Documentation

RDD Programming Guide

Section: "Basics" -> "Lazy Evaluation".

3. Book: Spark: The Definitive Guide (by Bill Chambers and Matei Zaharia): Chapter 2

"A Gentle Introduction to Spark

" explicitly details this behavior. It explains that transformations like filter() and groupBy() are lazy

and the logical plan is only executed when an action like count() is called.

Source: Chambers

B.

& Zaharia

M. (2018). Spark: The Definitive Guide. O'Reilly Media

Inc. Chapter 2

"The Concept of Lazy Evaluation" (pp. 26-27).