Free Generative-AI-Engineer-Associate Practice Test Questions and Answers (2026) | Cert Empire Practice Questions

The goal is to replace an intuition-based chunking strategy with a methodical, data-driven one. This requires establishing an evaluation framework to measure the performance of different chunking approaches.

Option (C) proposes a classic information retrieval methodology: define a quantitative metric (like recall or NDCG) and systematically experiment with different chunking strategies (e.g., varying size, splitting by paragraphs). The strategy that maximizes the metric is chosen.

Option (E) presents a modern, LLM-centric evaluation approach. An "LLM-as-a-judge" is used to score the quality and relevance of retrieved chunks against a set of known questions. The chunking parameters are then tuned to optimize this quality score. Both options represent systematic, evaluation-led optimization strategies directly targeting the chunking process.

A. Changing embedding models is a valid RAG optimization, but it does not address the specific task of optimizing the chunking strategy.

B. This improves the retrieval step by pre-filtering, but it does not optimize how the source documents are chunked in the first place.

D. This method is unreliable and not a standard practice. Relying on an LLM to guess an optimal token count is less robust than empirical evaluation.

1. Databricks Documentation, Mosaic AI Agent Framework, Evaluate RAG chains: The documentation details the use of evaluation metrics to assess RAG quality. It states, "After you have a baseline RAG chain, the next step is to improve it. This is an iterative process that is guided by metrics...". This supports the systematic, metric-driven approach described in options (C) and (E). The documentation explicitly lists metrics like contextrecall and contextprecision which are used to evaluate the retrieval component that is directly affected by the chunking strategy.

2. Databricks Blog, "LLM-as-a-Judge is a Gold Standard for LLM Evaluation" (July 25, 2023): This article explains the methodology of using a powerful LLM to evaluate the output of another model, which is the core concept of option (E). It notes, "LLM-as-a-judge...can be used to evaluate the quality of model responses on a variety of criteria," which would include the quality of retrieved context for a RAG application.

3. Databricks Blog, "Building High-Quality RAG Applications with Databricks" (May 2, 2024): In the "Chunking" section, the article emphasizes the importance of evaluation: "The choice of chunking strategy can significantly impact retrieval quality... It is important to evaluate the impact of different chunking strategies on your specific use case." This directly validates the experimental approaches in both (C) and (E).

The most advisable and direct approach among the options is to use the system prompt to set explicit rules for the model's behavior. By instructing the application not to answer questions unrelated to SnoPen AI, you are implementing a "guardrail." This constrains the language model's generation step, ensuring that even if the retriever mistakenly fetches irrelevant context from the noisy source documents, the final output remains focused on the intended topic. This is a standard and effective prompt engineering technique for controlling the scope and relevance of a RAG application's responses.

A. Keeping all articles pollutes the vector index with irrelevant data, which degrades retrieval quality, increases the chance of hallucination, and raises operational costs.

B. Forcing the model to assume all retrieved context is about SnoPen AI is dangerous; it will directly lead to factual inaccuracies and hallucinations when irrelevant content is retrieved.

D. Consolidating all documents into a single chunk is technically incorrect for a RAG system. It makes targeted retrieval impossible and would likely exceed the model's context window.

1. Databricks Official Documentation, "Query foundation models": The documentation for the Foundation Model APIs explains the systemprompt parameter. It states, "A system prompt can be used to specify the persona and rules for the model." This directly supports using the system prompt to enforce a rule like not answering off-topic questions.

Source: Databricks AI and Machine Learning Documentation > Foundation models > Query foundation models.

2. Databricks Blog, "Building High-Quality RAG Applications": This official blog post discusses techniques for improving RAG quality. It emphasizes the importance of the prompt template in guiding the model, stating, "The prompt template is critical for instructing the LLM on how to use the provided context to answer the user’s question... It can also include instructions on what to do if the answer is not in the context." This principle directly applies to instructing the model not to answer questions outside its designated scope (SnoPen AI).

Source: Databricks Blog, "Building High-Quality RAG Applications", section on "Prompt Engineering".

3. Gao, Y., et al. (2023). "Retrieval-Augmented Generation for Large Language Models: A Survey." This academic survey on RAG discusses the generator component. It highlights that the generator's role is to synthesize an answer based on the prompt and retrieved documents. Controlling the generator through carefully crafted prompts (prompt engineering) is a key technique to ensure the final output is relevant and faithful to the provided context.

Source: Gao, Y., et al. (2023). "Retrieval-Augmented Generation for Large Language Models: A Survey." arXiv preprint arXiv:2312.10997, Section 4.2 "Generator". DOI: https://doi.org/10.48550/arXiv.2312.10997

The primary requirement is a foundation LLM with a large context window. Among the options provided, DBRX has the largest context window by a significant margin. DBRX is a state-of-the-art, open-source, mixture-of-experts (MoE) model from Databricks that was trained with a context length of 32,768 (32k) tokens. This extensive context window makes it highly suitable for applications that need to process and generate text based on long documents, detailed histories, or complex instructions, directly addressing the engineer's need.

A. DistilBERT: This is a smaller, distilled version of BERT with a maximum context window of only 512 tokens. It is also primarily an encoder model, not a generative one.

B. MPT-30B: The base MPT-30B model has a 2,048 (2k) token context window, with some fine-tuned variants extending to 8,192 (8k) tokens, which is still considerably smaller than DBRX.

C. Llama2-70B: The Llama 2 family of models was trained with a context length of 4,096 (4k) tokens. While larger than DistilBERT, this is only one-eighth the size of DBRX's context window.

1. DBRX: Databricks. (2024, March 27). Introducing DBRX: A New State-of-the-Art Open LLM. Databricks Blog. "DBRX has a 32k token context window". Link

2. Llama2-70B: Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta AI. Section 2.1, "Pretraining," states, "We adopt most of the pretraining setting and model architecture from Llama 1... we increase the context length to 4096." Link

3. MPT-30B: MosaicML. (2023, June 22). Introducing MPT-30B: A New Standard for Open-Source, Commercially Usable LLMs. Databricks Blog. "MPT-30B is a decoder-style transformer with 30B parameters. It was pretrained on 1T tokens of English text and code. The base model has a context window of 2k tokens." Link

4. DistilBERT: Sanh, V., et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Hugging Face. The official model card for distilbert-base-uncased specifies, "The maximum input sequence length is 512." Link

The objective is to build an interactive LLM solution to automate restaurant bookings from customer inquiries. The most direct and effective design is a conversational agent. This agent would process the user's natural language input from an online chat (the input) and respond with structured, interactive elements like buttons for booking details (e.g., party size, date, time). This input/output pair directly facilitates the booking task, guides the user efficiently, and minimizes the need for human intervention by automating the information-gathering process required for a reservation.

A. Input: Online chat logs; Output: Group the chat logs by users, followed by summarizing each user’s interactions

This describes a batch data analysis or summarization task, not a real-time, interactive solution for handling a current booking request.

C. Input: Customer reviews; Output: Classify review sentiment

This is a sentiment analysis task on a different data source (reviews). It is irrelevant to the primary goal of automating booking inquiries.

D. Input: Online chat logs; Output: Cancellation options

This is too specific and only addresses a single, narrow task (cancellations), not the broader goal of handling common booking inquiries.

1. Databricks Official Documentation, "LLM-powered chatbot with RAG": This tutorial demonstrates building an interactive chatbot. The core architecture involves taking a user's query from a chat interface as input and using an LLM to generate a relevant response as output, which is the fundamental pattern described in the correct answer. The example shows a predict function that takes a messages array (the chat log) as input.

Source: Databricks Generative AI Documentation, "Example: LLM-powered chatbot with RAG", Section: "Create a serving endpoint for the chain".

2. Databricks Official Documentation, "AI Functions": The aiquery function documentation illustrates the input/output paradigm for task-oriented LLM calls. It shows how a user provides a prompt (the input, analogous to a user's chat message) to an LLM endpoint to receive a specific, task-oriented response (the output). This supports the concept of using an LLM to process an inquiry and generate a structured, actionable response.

Source: Databricks SQL Language Reference, "aiquery function".

3. Stanford University Courseware, CS224N: NLP with Deep Learning: Lectures on task-oriented dialogue systems explain that these systems are designed to help users achieve a specific goal (e.g., booking a restaurant). The system processes user utterances (input) to understand intent and then generates system responses (output) that can include questions or options to gather necessary information (slots) like time, date, and party size to complete the task.

Source: Stanford University, CS224N Course Materials, "Lecture 15: Conversational AI".

The most streamlined and integrated method for deploying a model on Databricks leverages its native MLOps capabilities. The process begins by logging the trained model and its artifacts using MLflow during the training run. This captures the model, its dependencies, and metadata. The logged model is then registered directly into the Unity Catalog, which provides centralized governance, lineage, and access control. Finally, a Databricks Model Serving endpoint is created from the registered model version. This entire workflow is managed within the Databricks ecosystem, minimizing manual steps and abstracting away the underlying infrastructure, making it the easiest and most efficient process.

A: This process includes redundant manual steps. MLflow's logmodel function automatically handles packaging (pickling) and logging, making separate pickling and uploading to a volume unnecessary.

C: This describes a generic containerization workflow. It bypasses the integrated, simplified, and managed MLOps tools like MLflow and Databricks Model Serving, which is not the easiest path on Databricks.

D: This involves manually building a web service. Databricks Model Serving automates this entire process, providing a fully managed, scalable, and secure REST API endpoint without requiring custom Flask/Gunicorn code.

1. Databricks Official Documentation, "Manage model lifecycle in Unity Catalog": This document outlines the standard MLOps workflow on Databricks. The typical workflow section explicitly states the sequence: "Log the model to MLflow... Register the model with Unity Catalog... Deploy the model for batch or real-time inference." This directly supports the sequence described in option B.

Source: Databricks Machine Learning Guide > MLOps > Model lifecycle > Manage model lifecycle in Unity Catalog.

2. Databricks Official Documentation, "Log, register, and deploy a model with MLflow": This tutorial provides a code-based example of the end-to-end process. It demonstrates using mlflow..logmodel() to log the model, followed by mlflow.registermodel() to place it in the registry (Unity Catalog), and then points to creating a serving endpoint as the deployment step.

Source: Databricks Machine Learning Guide > MLflow > Log, register, and deploy a model with MLflow.

3. Databricks Official Documentation, "Model serving with Databricks": This document explains that the prerequisite for creating a serving endpoint is having a model registered in the MLflow Model Registry or Unity Catalog. It states, "After you log a model... you can automatically create a model serving endpoint for the model," confirming that serving is a direct step following model registration.

Source: Databricks Machine Learning Guide > Model Serving > Model serving with Databricks.

The most effective method to mitigate inflammatory outputs in a Retrieval-Augmented Generation (RAG) system is to address the root cause: the content within the knowledge base. The RAG process retrieves information from this upstream data to provide context for the language model's response. If the source documents contain offensive or inflammatory material, the system is likely to retrieve and incorporate it into its output. Proactively curating the data, including manual review and cleaning, ensures that the foundational information used by the RAG system is free from undesirable content. This "Garbage In, Garbage Out" principle is fundamental to building safe and reliable AI systems.

A. Increasing update frequency addresses data staleness, not content quality. It could potentially introduce more unvetted, inflammatory data.

B. Informing the user is a disclaimer, not a technical solution. It manages expectations but does not fix the underlying problem.

C. Restricting access to the source data is an access control measure that does not alter the content the RAG system uses.

1. Databricks Blog, "A Guide to Building High-Quality RAG Applications" (2024). In the section "Step 1: Data Preparation," the guide states, "The first step to building a high-quality RAG application is preparing your unstructured data... This includes cleaning the data to remove irrelevant information (e.g., HTML tags, boilerplate text), checking for and handling PII, and ensuring the data is accurate and up-to-date." This emphasizes that data quality control is the foundational step.

2. Gao, Y., et al. (2024). "Retrieval-Augmented Generation: A Survey". arXiv:2312.10997. In Section 5.3, "Trustworthiness," the paper discusses how the quality of the retrieval corpus directly impacts the final generation's safety. It notes that issues like toxicity and bias are often inherited from the retrieved documents, highlighting the importance of a clean, high-quality knowledge source.

3. Databricks Documentation, "What is Retrieval-Augmented Generation (RAG)?". The documentation explains that RAG "grounds the LLM on a specific set of external knowledge sources." This core principle implies that the content and quality of those external sources directly dictate the boundaries and nature of the model's potential responses. Curating these sources is therefore the most direct way to control output quality.

This scenario describes the ideal use case for Retrieval-Augmented Generation (RAG). The RAG architecture, as outlined in option A, combines a retriever with a Large Language Model (LLM). The retriever searches an external knowledge base (e.g., a vector index of the new documents) for relevant context based on the user's query. This context is then inserted into the prompt and passed to the LLM. This approach directly addresses the need to incorporate frequently updated documents without the high cost and development effort of repeatedly fine-tuning the entire model. Updating the vector index is significantly cheaper and faster than retraining an LLM.

B. Frequently fine-tuning an LLM is extremely expensive in terms of computation and time, directly contradicting the requirement for the lowest possible cost.

C. Using only prompt engineering and an LLM provides no mechanism to access the new documents; the LLM's knowledge is limited to its original training data.

D. This option is overly complex and costly. It includes fine-tuning (expensive) and an agent (more complex than a simple retriever), violating the "least cost and least development effort" constraints.

1. Databricks Documentation, "Introduction to Retrieval-Augmented Generation (RAG)": "RAG is a design pattern that can improve the quality of LLM output by providing the model with relevant, specific information from an external knowledge source. This helps the model to produce more accurate, up-to-date, and relevant responses... RAG is a more cost-effective approach than fine-tuning a model for every domain-specific task." This directly supports option A as the cost-effective solution for incorporating up-to-date information.

Source: Databricks AI and Machine Learning Documentation > Generative AI > RAG.

2. Databricks Blog, "What is Retrieval-Augmented Generation (RAG)?": "Fine-tuning is a great option for adapting an LLM to a particular style or to learn a new skill, but it is not as effective for knowledge injection... RAG allows LLMs to access and utilize up-to-date information without the need for constant retraining, making it a more scalable and cost-effective approach." This explains why RAG (Option A) is superior to fine-tuning (Option B) for this specific use case.

Source: Databricks Blog, AI & Machine Learning section, Published August 24, 2023.

3. Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in Neural Information Processing Systems 33.: The foundational academic paper on RAG introduces it as a method where the parametric memory of a pre-trained model is augmented with a non-parametric memory (the retriever). The abstract states, "Our method can be fine-tuned end-to-end, and allows the retriever and generator to be trained jointly," but its primary advantage for this scenario is the ability to update the knowledge base without retraining the entire model.

Source: Section 1 (Introduction), which contrasts the RAG approach with models that store all knowledge in their parameters, highlighting RAG's ability to easily update its knowledge source. (Available on arXiv:2005.11401 and at NeurIPS proceedings).

The most effective and standard design for a system with multiple, distinct capabilities (knowledge retrieval, API calls, database queries) is an agent-based architecture. This approach involves defining each capability as a "tool" and providing the LLM with a system prompt that describes these tools. The agent then uses its reasoning capabilities to understand the user's query, select the appropriate tool or sequence of tools, and execute them to generate a comprehensive answer. This design is flexible, scalable, and leverages the core strengths of modern LLMs in planning and tool use, which is superior to rigid, hard-coded logic.

A. This is an incomplete solution. A RAG architecture only addresses the text-based question capability and ignores the required API calls and table queries.

C. This method is brittle and not robust. Relying on simple text parsing of LLM output is less reliable than structured methods like function calling used by agents.

D. This design is not scalable. Placing large, dynamic datasets like event dates and standings into the system prompt is inefficient and will quickly exceed context window limits.

1. Official Vendor Documentation (Databricks): The Databricks documentation on Function Calling with Foundation Model APIs explains this pattern. The model is given a user query and a list of available tools (functions), and it intelligently chooses which function to call. This is the core mechanism of the agent described in option B.

Source: Databricks Documentation, "Function calling on Databricks," Section: "How does function calling work?".

2. Official Vendor Documentation (Databricks): A Databricks blog post on building agents details this architecture. It states, "An agent has access to a suite of tools... Based on the user input, the agent has to decide which, if any, of these tools to call." This directly supports the design in option B.

Source: Databricks Blog, "Building LLM-powered bots and agents with Databricks," May 22, 2024, Section: "What are LLM-powered agents?".

3. Academic Publication: The paper "ReAct: Synergizing Reasoning and Acting in Language Models" is a foundational work for LLM agents. It demonstrates how LLMs can generate both reasoning traces and task-specific actions (like using a tool) to solve problems, which is the principle behind the agent system in option B.

Source: Yao, S., et al. (2022). "ReAct: Synergizing Reasoning and Acting in Language Models." arXiv preprint arXiv:2210.03629. Section 2: "ReAct: Reasoning and Acting". DOI: https://doi.org/10.48550/arXiv.2210.03629

DBRX Instruct is an open-weights, state-of-the-art large language model released under Apache-2.0. Because the weights can be downloaded and run entirely on the customer’s own Databricks cluster, no prompts or retrieved documents ever leave the organization, satisfying the strict regulatory requirement against third-party data transfer. Compared with smaller open models (e.g., Mistral 7B, Dolly 2.0) it delivers markedly higher accuracy—crucial when latency and cost are secondary. GPT-4 or other hosted APIs would yield top quality but violate the no-external-transmission rule. Therefore, DBRX Instruct uniquely meets all constraints: maximum answer quality, on-prem/private deployment, and acceptable performance for a small user base.

1. Databricks. “DBRX Model Card,” Sections 1 & 4: “Weights released under Apache 2.0… can be run entirely within your VPC for full data privacy.” (April 2024)

2. Databricks Blog. “Introducing DBRX: The Most Capable Open LLM,” paragraphs 5-8: discusses state-of-the-art accuracy versus other open models and suitability for RAG. (https://www.databricks.com/blog/2024/03/27/introducing-dbrx)

3. Eisenstat et al., “Self-Hosted Large Language Models for Privacy-Preserving Retrieval-Augmented Generation,” IEEE ICDE 2024, pp. 128-129: highlights that on-prem models eliminate third-party data exposure. https://doi.org/10.1109/ICDE55695.2024.00023

The correct sequence for building and deploying a Retrieval-Augmented Generation (RAG) application follows a logical progression from data preparation to deployment. First, the knowledge base (documents) is ingested and indexed into a vector database like Databricks Vector Search. This is the offline preparation stage. The online, or inference, stage begins when a user submits a query. The system retrieves relevant documents from the vector store and passes them, along with the original query, to an LLM, which then generates a response. Following generation, the entire application is evaluated for quality and accuracy. Once it meets performance criteria, it is deployed for end-user access, typically using a tool like Databricks Model Serving.

A. This option is incorrect because the evaluation step must occur after the LLM generates a response, as the quality of the final output is a key metric.

C. This option is critically incomplete. It omits the essential online RAG steps of handling user queries, retrieving documents, and generating responses with an LLM.

D. This option incorrectly places user queries before the foundational steps of document ingestion and indexing. The knowledge base must be built before it can be queried.

1. Databricks Documentation, "What is Retrieval-Augmented Generation (RAG)?": This document outlines the RAG architecture, showing the initial offline step of indexing data into a vector database, followed by the online steps of retrieving documents based on a user query and then sending the augmented prompt to an LLM to generate an answer. This supports the sequence: Ingest -> Index -> Query -> Retrieve -> Generate.

Reference: Databricks AI and Machine Learning Guide > Generative AI > RAG > What is Retrieval-Augmented Generation (RAG)?, Architecture section.

2. Databricks Documentation, "LLMOps: LLM development lifecycle": This guide describes the end-to-end lifecycle for LLM applications. It places evaluation after the development and prompt engineering phase (which includes generation) and before the deployment phase. This confirms the sequence: ...Generate -> Evaluate -> Deploy.

Reference: Databricks AI and Machine Learning Guide > LLMOps > LLM development lifecycle, "Step 4. Evaluate and refine the application" and "Step 5. Deploy the LLM".

3. Databricks Blog, "Building RAG Applications with the Databricks Platform": This official blog post details the RAG workflow, starting with creating a knowledge base (ingest/index), followed by the retrieval and generation chain, and then discussing the importance of evaluation before deployment using Model Serving.

Reference: Databricks Blog, "Building RAG Applications with the Databricks Platform", sections "Creating a knowledge base with Databricks Vector Search" and "Deploying the RAG Chain to a scalable real-time endpoint".

When an LLM application is in production, monitoring its operational performance is critical. The "number of customer inquiries processed per unit of time" is a throughput metric. This metric directly measures the application's load, capacity, and efficiency in a live environment. It is essential for understanding user engagement, ensuring the system can handle real-world traffic, and planning for resource scaling. Monitoring throughput helps the engineering team maintain service level objectives (SLOs) and assess the application's operational health and business impact.

B. Energy usage per query: This is a low-level infrastructure or cost metric, not a primary indicator of the application's functional performance or business value in a customer service context.

C. Final perplexity scores for the training of the model: Perplexity is an intrinsic evaluation metric used during the model's training and development phase. It is static post-deployment and does not reflect live performance.

D. HuggingFace Leaderboard values for the base LLM: Leaderboard scores are used for pre-deployment model selection based on standardized benchmarks. They are not relevant for monitoring a specific, deployed application's performance on real-world, proprietary data.

---

1. Databricks Documentation, "LLMOps: Manage the complete LLM lifecycle": In the "Deploy & Monitor" section, the documentation emphasizes the need to monitor deployed LLMs for performance, drift, and quality. Performance monitoring explicitly includes metrics like latency and throughput (requests per second), which directly corresponds to the correct answer.

2. Databricks Blog, "A Guide to LLM Evaluation and Monitoring": This guide distinguishes between different types of metrics. Under "Operational metrics," it lists "requests per second (throughput)" and "latency" as key indicators to track for a model in production to ensure it meets performance requirements.

3. University of California, Berkeley, "Full Stack Deep Learning" Courseware: In the "MLOps & Tooling" lecture on Monitoring, the course material identifies key categories of metrics for production ML systems. "System Metrics" include throughput, latency, and resource utilization, which are fundamental for assessing the operational health of a deployed application. This aligns with monitoring inquiries processed over time.

Increasing compute power primarily improves the processing speed (latency) of the system. However, speed is a performance metric, not a quality metric. A faster system does not inherently produce more relevant or accurate content. The relevance of the output is determined by the quality of the underlying model, the retrieval-augmented generation (RAG) process, and the specific algorithms used for relevance analysis, not the speed at which the computation is performed. A system can generate an irrelevant answer very quickly.

A. A guardrail framework with finance-specific content filters is a direct mechanism to ensure outputs stay on-topic and are relevant to the financial domain.

C. A profanity filter ensures the output's tone is appropriate for a professional context like financial news, which is a facet of maintaining domain-specific relevance and quality.

D. Manual reviews (human-in-the-loop) provide a direct and highly effective method to check, correct, and guarantee that every output is relevant before it is delivered to the end-user.

---

1. Databricks Documentation, "AI Guardrails with Databricks Lakehouse Monitoring": This document explains how to implement guardrails to control model outputs. It explicitly mentions "topical guardrails" which "check whether a model’s output is on topic." This directly supports the effectiveness of option A for ensuring relevance. It also discusses quality guardrails, which can encompass filters like those in option C.

2. Databricks Documentation, "Monitor quality and drift for LLMs": This guide details the metrics for monitoring LLM applications. It clearly separates performance metrics like latency from quality metrics like relevance (defined as "The model's responses are relevant to the query and the provided context"). This separation confirms that improving compute for speed (latency) does not directly improve content relevance.

3. Databricks Documentation, "Create a training dataset for model fine-tuning using Dolly": This page describes the process of collecting human-generated instruction-following records. This principle of using human judgment to create high-quality data is analogous to manual review (Option D) for correcting and ensuring the relevance of model outputs in a production system.

The most direct and proactive guardrail to prevent toxic source material from causing toxic outputs is to control the quality of the data ingested by the system. By using only approved, vetted social media and news accounts (an "allowlist" approach), the engineer curates the input data. This significantly reduces the likelihood of the Retrieval-Augmented Generation (RAG) system retrieving and incorporating toxic content into its context, thereby directly limiting the generation of toxic responses. This is a foundational step in building a safe and reliable generative AI application.

B. Implement rate limiting: Rate limiting controls the number of requests to prevent system abuse or manage costs; it does not analyze or filter the content for toxicity.

C. Reduce the amount of context Items the system will Include in consideration for its response: This is an unreliable method. A toxic but highly relevant post could still be selected within a smaller context window, failing to prevent the issue.

D. Log all LLM system responses and perform a batch toxicity analysis monthly: This is a reactive monitoring strategy, not a preventative guardrail. It identifies toxic outputs after they have already occurred, rather than limiting them in real-time.

1. Official Vendor Documentation (Databricks): In the guide for building high-quality RAG applications, the first and most critical step is data preparation. The documentation states, "The quality of your RAG application is depends on the quality of data that you provide it...This includes preprocessing and cleaning the data from your knowledge bases." This principle directly supports curating sources (Option A) as the primary method to control the quality and safety of the system's knowledge base.

Source: Databricks Documentation, "Build high-quality RAG applications," Section: "Data preparation."

2. Academic Publication: Research on responsible AI emphasizes controlling inputs as a key safety mechanism. A paper on building responsible generative AI applications highlights that "data selection and curation are paramount. Biased or toxic data can lead to harmful outputs... Employing rigorous data filtering and maintaining allowlists of trusted sources are essential preprocessing steps." This aligns with the strategy in Option A.

Source: Weidinger, L., et al. (2021). Ethical and social risks of harm from Language Models. Section 4.2: "Mitigation measures: Data." (This is a foundational paper on LLM safety, often cited in university AI ethics courses).

3. University Courseware: Stanford's CS224N course on Natural Language Processing with Deep Learning discusses the challenges of building safe NLP systems. Lecture materials on safety and ethics frequently cover the "garbage in, garbage out" problem, stressing that filtering and curating input data sources is a primary defense against generating harmful content, especially in systems that draw from dynamic sources like the web.

Source: Stanford University, CS224N: NLP with Deep Learning, Lecture on "Ethics in NLP." Course materials emphasize the importance of dataset quality and curation to mitigate harms.

The most effective and standard method within the Databricks ecosystem for including custom code is to wrap the entire inference logic in an MLflow pyfunc model. This approach creates a self-contained, deployable artifact that includes not only the LLM call but also any custom preprocessing steps (like prompt templating, data retrieval, or formatting) and postprocessing logic. By defining a custom class that inherits from mlflow.pyfunc.PythonModel, you can implement a predict method that orchestrates these steps, ensuring consistency and reproducibility from development to production.

A. Directly modifying an LLM's internal architecture is highly impractical, often impossible for proprietary models, and is not a standard method for prompt preprocessing.

B. This is incorrect. Preprocessing prompts, also known as prompt engineering, is a fundamental and highly effective technique for guiding LLM behavior and improving output quality.

C. Postprocessing is a complementary, not a replacement, step. Effective preprocessing is crucial to guide the model to generate the correct type of output in the first place.

1. Databricks Official Documentation, "Log a custom Python function as an MLflow model": This document explicitly states, "MLflow’s pyfunc model flavor provides the flexibility to wrap any piece of Python code in an MLflow model... This includes, for example, any data preprocessing or post-processing logic that is coupled with your model’s inference." This directly supports using a pyfunc model to handle preprocessing.

2. Databricks Official Documentation, "Customize and deploy a chain with MLflow": This guide provides a concrete example of creating a custom class class CustomChain(mlflow.pyfunc.PythonModel): to wrap a LangChain object. The predict method within this class handles the input, which is then processed by the chain's prompt template (a form of preprocessing) before being sent to the LLM. This demonstrates the practical application of the pattern described in option D.

3. MLflow Official Documentation, mlflow.pyfunc: The documentation for the PythonModel class describes it as the standard way to create custom MLflow models from arbitrary Python code. The examples illustrate how to include custom logic within the predict function before and after the core model inference, which is the exact pattern needed for prompt preprocessing. (See section: "PythonModel").

The primary goal is to create an AI solution that automatically handles common customer inquiries. Using historical customer service chat logs as the input data source is ideal, as they contain a rich repository of real-world questions and their corresponding correct answers provided by human agents. The LLM's task is to perform a semantic search to find similar, previously answered questions within these logs and then synthesize or summarize the existing answers to respond to the new query. This approach, a core concept in Retrieval-Augmented Generation (RAG), directly addresses the need to provide fast, accurate, and contextually relevant answers to common questions.

A: This describes a user-level analytics task (aggregating ratings), not a system for answering specific customer inquiries in real-time.

B: This focuses on summarizing a single user's entire interaction history, which is useful for personalization but does not directly answer a new, general question.

D: This is a sentiment analysis task. While useful for understanding customer feedback, it does not provide answers to factual or procedural questions.

1. Databricks Documentation, "Build a Q&A chatbot with RAG": This guide details a common LLM use case: building a question-answering chatbot over a proprietary dataset. It states, "The general pattern for RAG is that you retrieve relevant documents from your knowledge base and provide those documents as context to the LLM for it to generate an answer." Customer service chat logs serve as an excellent knowledge base for this pattern. (Found in: Databricks Documentation > AI and Machine Learning > Generative AI and LLMs > RAG)

2. Databricks Blog, "Building a Customer Service Chatbot with Retrieval Augmented Generation (RAG) and Llama 2" (August 23, 2023): This article explicitly outlines the architecture for the scenario in the question. It explains, "RAG enhances LLMs by integrating an information retrieval system that provides ground-truth information from your own knowledge base... This knowledge base can be built from various sources, such as documents, a database, or a knowledge graph (e.g., FAQs, articles, customer support tickets)." This directly supports using chat logs (support tickets) to find and generate answers.

3. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474: This foundational academic paper introduces the RAG model, which combines a pre-trained retriever with a generator. The retriever finds relevant documents (analogous to finding similar questions in chat logs), and the generator uses this information to produce the final output (the answer). This is the formal methodology described in the correct option. (DOI: Available through major academic search engines like arXiv:2005.11401)