The scenario describes an attacker crafting specific inputs to bypass an LLM's built-in restrictions and manipulate its output. This technique is known as prompt injection. The attacker injects malicious instructions within the prompt, causing the model to ignore its original system-level instructions and follow the attacker's commands instead. This directly exploits the model's interpretation of input to compromise its output integrity, making it the most accurate description of the attack.

B. Jailbreaking is a specific goal or outcome of a prompt injection attack, aimed at bypassing safety and ethical filters, rather than the attack technique itself.

C. Remote code execution involves executing arbitrary code on the server, a more severe and distinct attack that is not inherently part of manipulating LLM text output.

D. Evasion is an adversarial attack that typically causes a model (e.g., a classifier) to make an incorrect prediction, which is different from overriding an LLM's core instructions.

1. OWASP Foundation. (2023). OWASP Top 10 for Large Language Model Applications. In the "LLM01: Prompt Injection" section

the vulnerability is defined as: "Prompt injection vulnerabilities allow a malicious user to manipulate the output of a Large Language Model (LLM) through crafted inputs... This can lead to data exfiltration

unauthorized access

or other security breaches."

2. National Institute of Standards and Technology (NIST). (2023). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST.AI.100-2 E2023). Section 3.2.1

"Data/Input Poisoning

" describes how attackers can manipulate inputs. Prompt injection is cited as a prime example for LLMs where "an attacker can carefully craft a prompt to an LLM to have it ignore previous instructions."

3. Greshake

K.

Abdelnabi

S.

Mishra

S.

Endres

C.

Holz

T.

& Fritz

M. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv preprint arXiv:2302.12173. Section 2

"Background

" defines prompt injection as an attack where an "adversary controls a part of a prompt... to manipulate the LLM's behavior." (https://doi.org/10.48550/arXiv.2302.12173)