Stratified evaluation is a robust methodology for analyzing complex systems with multiple variables. By segmenting the data into meaningful strata (agent variants, query complexity, time), it ensures that comparisons are fair and not skewed by confounding factors. This approach isolates variables, allowing for precise identification of performance differences. Tracking decision paths provides qualitative data on why an agent succeeded or failed, directly enabling root cause analysis. This systematic approach is superior to simple averaging or limited comparisons, providing a comprehensive view for targeted optimization.

A. This approach introduces selection bias by focusing only on improved agents, ignoring the systemic issues that may be present in underperforming ones.

B. Averaging metrics masks the very inconsistencies that need to be analyzed, making it impossible to identify specific root causes or outliers.

D. While comparing top and bottom performers is useful, it is less systematic than stratified sampling and may miss nuances related to query type or time.

1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. (Chapter 18 discusses stratified sampling to reduce variance and improve the precision of evaluation in large-scale systems).

2. NVIDIA AI Enterprise Documentation. (2023). MLOps: Managing the AI Lifecycle. Section on Model Evaluation. (Discusses the principle of rigorous, segmented evaluation to understand model performance across different data slices before deployment and for ongoing monitoring).

3. Stanford University. (CS 229: Machine Learning). Course Notes on Model Evaluation. (Explains the importance of evaluating models on representative, and sometimes stratified, subsets of data to diagnose performance issues).