The total number of models to be trained is determined by multiplying the number of hyperparameter combinations by the number of cross-validation folds.

First, calculate the number of hyperparameter combinations from the grid search:

(Values for Hyperparameter 1) × (Values for Hyperparameter 2) = 3 × 2 = 6 combinations.

Next, for each of these 6 combinations, 3-fold cross-validation is performed. This requires training a separate model for each fold.

Total models = 6 combinations × 3 folds = 18 models.

Since each of these 18 model training runs is an independent task, a distributed computing platform like Databricks can execute all of them in parallel, assuming sufficient cluster resources are available.

A. 3: This represents only the number of cross-validation folds, not the total number of models trained across all hyperparameter combinations.

B. 5: This is the sum of the number of hyperparameter values (3 + 2), which is an incorrect calculation for a grid search.

C. 6: This correctly identifies the number of hyperparameter combinations (3 × 2) but omits the 3 models trained for each combination due to 3-fold cross-validation.

---

1. Apache Spark Official Documentation

pyspark.ml.tuning.CrossValidator: The documentation for Spark's CrossValidator describes the process: "For each paramMap

CrossValidator will split the dataset into k folds. Then it will train on k-1 folds and evaluate on the remaining fold." This confirms that for each hyperparameter combination (a paramMap)

k models are trained (one for each fold). The parallelism parameter further confirms that these model fits can be executed in parallel. In this scenario

there are 6 paramMaps and k=3

resulting in 18 total model fits that can be parallelized.

Source: Apache Spark 3.5.0 Documentation

MLlib: Main Guide > ML Tuning: model selection and hyperparameter tuning > Cross-Validation.

2. Hastie

T.

Tibshirani

R.

& Friedman

J. (2009). The Elements of Statistical Learning: Data Mining

Inference

and Prediction. Springer. In Chapter 7

Section 7.10.1 "Cross-Validation

" the authors describe K-fold cross-validation. The process involves fitting the model K times on different subsets of the training data. When combined with a grid search

this fitting process is repeated for every point in the hyperparameter grid. The independence of each model fit makes the overall process highly parallelizable.

Source: Chapter 7

"Model Assessment and Selection

" Section 7.10.1

page 242.

3. Databricks Machine Learning Documentation

"Hyperparameter tuning": The documentation explains how tools like Hyperopt with SparkTrials can "distribute runs and manage models" for hyperparameter tuning. This distribution of runs across a cluster's worker nodes is the mechanism that enables the parallel training of the multiple models generated by a grid search and cross-validation process.

Source: Databricks Documentation > Machine Learning > Models > Hyperparameter tuning.