The best primary health metric to catch early regressions is the Task success rate. This metric directly measures the agent's ability to correctly and completely fulfill its intended purpose. Since the team has a daily labeled evaluation set, they can run automated tests to calculate this rate frequently. This provides a direct, objective, and timely signal of performance degradation, allowing the team to identify and fix regressions before they significantly impact users, thus protecting the accuracy SLO.

B. CSAT is a lagging indicator of user perception, not a direct or timely measure of functional accuracy needed to catch early regressions.

C. Token spend is a cost metric. It does not measure the agent's accuracy or effectiveness in completing tasks for users.

D. Total chats is a volume metric, not a quality or health metric. A failing agent could even cause an increase in chats as users retry.

1. Microsoft Learn. (2024). Analyze Copilot Studio bot performance and usage. Section: "Analytics > Session outcomes". The documentation highlights "Resolved rate" and "Abandon rate" as key performance indicators. Task success rate is a more general term for these core metrics that measure if the bot achieved its goal.

2. Lowe, R., et al. (2017). Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. (DOI: https://doi.org/10.18653/v1/P17-1103). This academic paper discusses the importance of automated, objective metrics for evaluating conversational AI performance, which aligns with using a labeled dataset to measure success.