The operational cost of large language models (LLMs) used in agents is primarily determined by the number of tokens processed. This includes tokens in the input prompt (model input size), tokens in the generated response (model output size), and tokens consumed by any integrated tools. By monitoring token usage, specifically metrics like prompttokens, completiontokens, and totaltokens, you can directly identify which component is responsible for the increased consumption and, consequently, the higher costs. This provides a precise, quantifiable measure to diagnose the financial impact of the recent update.

A. evaluation metrics: These metrics (e.g., relevance, groundedness) assess the quality and performance of the model's output, not the resource consumption or cost associated with generating it.

C. latency: Latency measures the response time of the agent. While larger inputs/outputs can increase latency, it is a performance metric, not a direct measure of operational cost.

D. run success rate: This metric indicates the reliability of the agent by tracking successful versus failed executions. It does not provide insight into the cost drivers of the successful runs.

---

1. Azure Machine Learning Documentation - Monitor prompt flow endpoints in production: This official document lists the key metrics collected for prompt flow endpoints. It explicitly mentions prompttokens, completiontokens, and totaltokens as available metrics for understanding consumption.

Reference: Microsoft. (n.d.). Monitor and collect data from prompt flow endpoints in production. Azure Machine Learning documentation. Retrieved from the official Microsoft Learn website. In the section "View metrics in Azure portal," see the table "Available metrics for prompt flow endpoints," which lists token-related metrics.

2. Azure OpenAI Service Pricing: The official pricing model for Azure OpenAI Service directly links cost to token consumption. The pricing is specified "per 1,000 tokens" for both prompt and completion, which confirms that token usage is the fundamental unit of cost.

Reference: Microsoft. (n.d.). Azure OpenAI Service - Pricing. Microsoft Azure. Retrieved from the official Microsoft Azure pricing page. Review the pricing tables for models like GPT-4 or GPT-3.5-Turbo, which show distinct costs for "Input" and "Output" tokens.

3. Azure Machine Learning Documentation - How to evaluate a prompt flow: This document details the built-in and custom evaluation metrics available in Prompt Flow. It lists metrics like "Groundedness," "Relevance," and "Coherence," demonstrating that their purpose is to measure quality, not cost.

Reference: Microsoft. (n.d.). How to evaluate a prompt flow. Azure Machine Learning documentation. Retrieved from the official Microsoft Learn website. See the section "Evaluation metrics."