What does in-context learning in Large Language Models involve?

In-context learning is a core capability of Large Language Models (LLMs) where the model learns to perform a new task based solely on information provided within the prompt at inference time. This is achieved by "conditioning" the model with a natural language description of the task and/or a few examples (demonstrations). The model uses this context to understand the desired pattern or format and generates a relevant completion without any updates to its underlying weights or parameters. This process is also referred to as few-shot, one-shot, or zero-shot prompting, depending on the number of examples provided.

A. Pretraining the model on a specific domain is a form of fine-tuning or continued pretraining, which permanently modifies the model's weights.

B. Training the model using reinforcement learning (e.g., RLHF) is a distinct training phase to align the model's behavior, not an inference-time learning method.

D. Adding more layers to the model is a change to the model's architecture and is unrelated to how it learns from a prompt during inference.

1. Brown, T. B., et al. (2020). "Language Models are Few-Shot Learners." Advances in neural information processing systems, 33, 1877-1901. In Section 2, "Few-Shot, One-Shot, and Zero-Shot Settings," the paper defines in-context learning as the model's ability to learn a task from a few examples provided in the context (the prompt) at inference time, without any gradient updates.

2. Oracle Cloud Infrastructure (OCI) Documentation. "Generative AI - Prompting basics." The documentation explains prompt engineering techniques, stating: "You can provide a few examples of input text and expected output in the prompt before the final input that you want the model to complete. This technique is called few-shot prompting." This is a direct application of in-context learning.

3. Stanford University, CS324: Large Language Models (Winter 2023). "Lecture 2: In-Context Learning." The course materials define in-context learning as a "paradigm for 'learning' from a small number of examples provided in the prompt" and explicitly contrast it with traditional training methods that update model parameters.