The summary() method on a PySpark DataFrame is specifically designed to compute a rich set of aggregate statistics. For numerical columns, it calculates the count, mean, standard deviation, minimum, maximum, and approximate quartiles (25%, 50%, 75%). The interquartile range (IQR) is the difference between the 75th and 25th percentiles, which are both provided by the output of the summary() function. Therefore, this single command directly provides all the information requested by the data scientist.

B. sparkdf.stats(): This is not a valid method for a Spark DataFrame. The correct attribute to access statistical functions is .stat, not .stats().

C. sparkdf.describe().head(): The .describe() method computes count, mean, standard deviation, min, and max, but it does not compute the percentiles required to find the interquartile range (IQR).

D. sparkdf.printSchema(): This method only displays the schema of the DataFrame (column names and data types) and does not compute any summary statistics.

E. sparkdf.toPandas(): This action converts the Spark DataFrame into a pandas DataFrame. It does not compute the statistics itself; it is a data collection step that precedes any analysis.

1. Apache Spark 3.5.0 Documentation

pyspark.sql.DataFrame.summary: "Computes specified statistics for numeric and string columns... If no statistics are given

this function computes count

mean

stddev

min

approximate quartiles (25%

50%

75%)

and max." This confirms that summary() provides the necessary quartiles for IQR.

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html

2. Apache Spark 3.5.0 Documentation

pyspark.sql.DataFrame.describe: "Computes basic statistics for numeric and string columns... For numeric columns

the result includes count

mean

stddev

min

max." This source verifies that .describe() lacks the required percentile information.

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.describe.html

3. Databricks Documentation

DataFrames: "You can use describe to see summary statistics for a DataFrame... To see more statistics

including quartiles

use the summary method." This official Databricks source explicitly differentiates between the two methods and highlights summary() for calculating quartiles.

Source: Databricks Documentation > Get started > DataFrames > Python > Summarize and visualize data. (Specific page URLs change

but the content is consistently found in the introductory DataFrame tutorials).