When an LLM application is in production, monitoring its operational performance is critical. The "number of customer inquiries processed per unit of time" is a throughput metric. This metric directly measures the application's load, capacity, and efficiency in a live environment. It is essential for understanding user engagement, ensuring the system can handle real-world traffic, and planning for resource scaling. Monitoring throughput helps the engineering team maintain service level objectives (SLOs) and assess the application's operational health and business impact.

B. Energy usage per query: This is a low-level infrastructure or cost metric, not a primary indicator of the application's functional performance or business value in a customer service context.

C. Final perplexity scores for the training of the model: Perplexity is an intrinsic evaluation metric used during the model's training and development phase. It is static post-deployment and does not reflect live performance.

D. HuggingFace Leaderboard values for the base LLM: Leaderboard scores are used for pre-deployment model selection based on standardized benchmarks. They are not relevant for monitoring a specific, deployed application's performance on real-world, proprietary data.

---

1. Databricks Documentation, "LLMOps: Manage the complete LLM lifecycle": In the "Deploy & Monitor" section, the documentation emphasizes the need to monitor deployed LLMs for performance, drift, and quality. Performance monitoring explicitly includes metrics like latency and throughput (requests per second), which directly corresponds to the correct answer.

2. Databricks Blog, "A Guide to LLM Evaluation and Monitoring": This guide distinguishes between different types of metrics. Under "Operational metrics," it lists "requests per second (throughput)" and "latency" as key indicators to track for a model in production to ensure it meets performance requirements.

3. University of California, Berkeley, "Full Stack Deep Learning" Courseware: In the "MLOps & Tooling" lecture on Monitoring, the course material identifies key categories of metrics for production ML systems. "System Metrics" include throughput, latency, and resource utilization, which are fundamental for assessing the operational health of a deployed application. This aligns with monitoring inquiries processed over time.

Increasing compute power primarily improves the processing speed (latency) of the system. However, speed is a performance metric, not a quality metric. A faster system does not inherently produce more relevant or accurate content. The relevance of the output is determined by the quality of the underlying model, the retrieval-augmented generation (RAG) process, and the specific algorithms used for relevance analysis, not the speed at which the computation is performed. A system can generate an irrelevant answer very quickly.

A. A guardrail framework with finance-specific content filters is a direct mechanism to ensure outputs stay on-topic and are relevant to the financial domain.

C. A profanity filter ensures the output's tone is appropriate for a professional context like financial news, which is a facet of maintaining domain-specific relevance and quality.

D. Manual reviews (human-in-the-loop) provide a direct and highly effective method to check, correct, and guarantee that every output is relevant before it is delivered to the end-user.

---

1. Databricks Documentation, "AI Guardrails with Databricks Lakehouse Monitoring": This document explains how to implement guardrails to control model outputs. It explicitly mentions "topical guardrails" which "check whether a model’s output is on topic." This directly supports the effectiveness of option A for ensuring relevance. It also discusses quality guardrails, which can encompass filters like those in option C.

2. Databricks Documentation, "Monitor quality and drift for LLMs": This guide details the metrics for monitoring LLM applications. It clearly separates performance metrics like latency from quality metrics like relevance (defined as "The model's responses are relevant to the query and the provided context"). This separation confirms that improving compute for speed (latency) does not directly improve content relevance.

3. Databricks Documentation, "Create a training dataset for model fine-tuning using Dolly": This page describes the process of collecting human-generated instruction-following records. This principle of using human judgment to create high-quality data is analogous to manual review (Option D) for correcting and ensuring the relevance of model outputs in a production system.

The most direct and proactive guardrail to prevent toxic source material from causing toxic outputs is to control the quality of the data ingested by the system. By using only approved, vetted social media and news accounts (an "allowlist" approach), the engineer curates the input data. This significantly reduces the likelihood of the Retrieval-Augmented Generation (RAG) system retrieving and incorporating toxic content into its context, thereby directly limiting the generation of toxic responses. This is a foundational step in building a safe and reliable generative AI application.

B. Implement rate limiting: Rate limiting controls the number of requests to prevent system abuse or manage costs; it does not analyze or filter the content for toxicity.

C. Reduce the amount of context Items the system will Include in consideration for its response: This is an unreliable method. A toxic but highly relevant post could still be selected within a smaller context window, failing to prevent the issue.

D. Log all LLM system responses and perform a batch toxicity analysis monthly: This is a reactive monitoring strategy, not a preventative guardrail. It identifies toxic outputs after they have already occurred, rather than limiting them in real-time.

1. Official Vendor Documentation (Databricks): In the guide for building high-quality RAG applications, the first and most critical step is data preparation. The documentation states, "The quality of your RAG application is depends on the quality of data that you provide it...This includes preprocessing and cleaning the data from your knowledge bases." This principle directly supports curating sources (Option A) as the primary method to control the quality and safety of the system's knowledge base.

Source: Databricks Documentation, "Build high-quality RAG applications," Section: "Data preparation."

2. Academic Publication: Research on responsible AI emphasizes controlling inputs as a key safety mechanism. A paper on building responsible generative AI applications highlights that "data selection and curation are paramount. Biased or toxic data can lead to harmful outputs... Employing rigorous data filtering and maintaining allowlists of trusted sources are essential preprocessing steps." This aligns with the strategy in Option A.

Source: Weidinger, L., et al. (2021). Ethical and social risks of harm from Language Models. Section 4.2: "Mitigation measures: Data." (This is a foundational paper on LLM safety, often cited in university AI ethics courses).

3. University Courseware: Stanford's CS224N course on Natural Language Processing with Deep Learning discusses the challenges of building safe NLP systems. Lecture materials on safety and ethics frequently cover the "garbage in, garbage out" problem, stressing that filtering and curating input data sources is a primary defense against generating harmful content, especially in systems that draw from dynamic sources like the web.

Source: Stanford University, CS224N: NLP with Deep Learning, Lecture on "Ethics in NLP." Course materials emphasize the importance of dataset quality and curation to mitigate harms.

The most effective and standard method within the Databricks ecosystem for including custom code is to wrap the entire inference logic in an MLflow pyfunc model. This approach creates a self-contained, deployable artifact that includes not only the LLM call but also any custom preprocessing steps (like prompt templating, data retrieval, or formatting) and postprocessing logic. By defining a custom class that inherits from mlflow.pyfunc.PythonModel, you can implement a predict method that orchestrates these steps, ensuring consistency and reproducibility from development to production.

A. Directly modifying an LLM's internal architecture is highly impractical, often impossible for proprietary models, and is not a standard method for prompt preprocessing.

B. This is incorrect. Preprocessing prompts, also known as prompt engineering, is a fundamental and highly effective technique for guiding LLM behavior and improving output quality.

C. Postprocessing is a complementary, not a replacement, step. Effective preprocessing is crucial to guide the model to generate the correct type of output in the first place.

1. Databricks Official Documentation, "Log a custom Python function as an MLflow model": This document explicitly states, "MLflow’s pyfunc model flavor provides the flexibility to wrap any piece of Python code in an MLflow model... This includes, for example, any data preprocessing or post-processing logic that is coupled with your model’s inference." This directly supports using a pyfunc model to handle preprocessing.

2. Databricks Official Documentation, "Customize and deploy a chain with MLflow": This guide provides a concrete example of creating a custom class class CustomChain(mlflow.pyfunc.PythonModel): to wrap a LangChain object. The predict method within this class handles the input, which is then processed by the chain's prompt template (a form of preprocessing) before being sent to the LLM. This demonstrates the practical application of the pattern described in option D.

3. MLflow Official Documentation, mlflow.pyfunc: The documentation for the PythonModel class describes it as the standard way to create custom MLflow models from arbitrary Python code. The examples illustrate how to include custom logic within the predict function before and after the core model inference, which is the exact pattern needed for prompt preprocessing. (See section: "PythonModel").

The primary goal is to create an AI solution that automatically handles common customer inquiries. Using historical customer service chat logs as the input data source is ideal, as they contain a rich repository of real-world questions and their corresponding correct answers provided by human agents. The LLM's task is to perform a semantic search to find similar, previously answered questions within these logs and then synthesize or summarize the existing answers to respond to the new query. This approach, a core concept in Retrieval-Augmented Generation (RAG), directly addresses the need to provide fast, accurate, and contextually relevant answers to common questions.

A: This describes a user-level analytics task (aggregating ratings), not a system for answering specific customer inquiries in real-time.

B: This focuses on summarizing a single user's entire interaction history, which is useful for personalization but does not directly answer a new, general question.

D: This is a sentiment analysis task. While useful for understanding customer feedback, it does not provide answers to factual or procedural questions.

1. Databricks Documentation, "Build a Q&A chatbot with RAG": This guide details a common LLM use case: building a question-answering chatbot over a proprietary dataset. It states, "The general pattern for RAG is that you retrieve relevant documents from your knowledge base and provide those documents as context to the LLM for it to generate an answer." Customer service chat logs serve as an excellent knowledge base for this pattern. (Found in: Databricks Documentation > AI and Machine Learning > Generative AI and LLMs > RAG)

2. Databricks Blog, "Building a Customer Service Chatbot with Retrieval Augmented Generation (RAG) and Llama 2" (August 23, 2023): This article explicitly outlines the architecture for the scenario in the question. It explains, "RAG enhances LLMs by integrating an information retrieval system that provides ground-truth information from your own knowledge base... This knowledge base can be built from various sources, such as documents, a database, or a knowledge graph (e.g., FAQs, articles, customer support tickets)." This directly supports using chat logs (support tickets) to find and generate answers.

3. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474: This foundational academic paper introduces the RAG model, which combines a pre-trained retriever with a generator. The retriever finds relevant documents (analogous to finding similar questions in chat logs), and the generator uses this information to produce the final output (the answer). This is the formal methodology described in the correct option. (DOI: Available through major academic search engines like arXiv:2005.11401)