The spark.executor.cores configuration property specifies the number of CPU cores allocated to each executor process. In Spark's architecture, each core within an executor can run one task at a time. Therefore, this setting directly controls the maximum number of tasks that can be executed concurrently on a single executor, defining its parallel processing capacity. For instance, setting spark.executor.cores to 4 allows an executor to run up to four tasks simultaneously.

B. spark.task.maxFailures: This property defines the number of times a task can be retried upon failure before the entire job is aborted; it relates to fault tolerance, not parallelism.

C. spark.executor.memory: This sets the total amount of heap memory available to an executor. While crucial for task execution, it does not determine the number of concurrent tasks.

D. spark.sql.shuffle.partitions: This configuration controls the number of partitions for data shuffling during wide transformations, which influences the number of tasks in a stage, not the concurrency within a single executor.

---

1. Official Apache Spark Documentation: The documentation explicitly states the purpose of spark.executor.cores.

Source: Apache Spark 3.4.1 Documentation

Configuration Page

Application Properties section.

Reference: Under the property spark.executor.cores

the description reads: "The number of cores to use on each executor. In standalone and YARN mode

this controls the number of concurrent tasks an executor can run."

2. Official Databricks Documentation: Databricks documentation reinforces this concept in its cluster configuration guides.

Source: Databricks Documentation

Cluster configuration best practices.

Reference: In discussions about cluster sizing and tuning

it is explained that the number of executor cores determines the number of task slots available per executor

directly impacting parallelism. For example

the "Worker type" section often links the number of vCPUs to the default spark.executor.cores setting.

3. University Courseware: Academic materials on distributed computing with Spark confirm this fundamental architectural principle.

Source: UC Berkeley

CS 186/286

Introduction to Database Systems (Fall 2018)

Lecture 20: Spark.

Reference: Slide 25 ("Anatomy of a Spark Job") illustrates that an Executor contains multiple "Task Slots" (cores)

and each slot runs one task. The number of these slots is configured by spark.executor.cores.