The Instruction Hierarchy: Teaching Models to Resist Prompt Injection

The Problem of Instruction Ambiguity

Large language models process all text in their context window as a flat sequence of tokens. There is no built-in mechanism to distinguish between instructions from the system operator (the system prompt), instructions from the user (the user message), and data that the model should process without treating as instructions (retrieved documents, tool outputs, web page content). This flat hierarchy is the root cause of prompt injection: any text in the context can potentially override any other text.

The instruction hierarchy approach, documented in OpenAI's research and increasingly adopted across the industry, addresses this by training models to recognize and enforce a privilege ordering. System-level instructions take precedence over user instructions, which take precedence over third-party content. When instructions at different privilege levels conflict, the model follows the higher-privilege instruction.

Training the Hierarchy

Implementing instruction hierarchy requires fine-tuning the model on adversarial examples where lower-privilege inputs attempt to override higher-privilege instructions. The training data includes scenarios where user messages instruct the model to ignore its system prompt, where retrieved documents contain instructions that conflict with user requests, and where tool outputs attempt to redirect the model's behavior.

The model learns to recognize these override attempts and refuse them. Importantly, the training preserves the model's ability to follow legitimate instructions at all privilege levels. A system prompt that says 'answer questions about cooking' should not prevent the model from following a user's request to 'explain how to make sourdough bread,' even though the user instruction could be viewed as attempting to control the model's behavior.

Limitations and Complementary Defenses

Instruction hierarchy is not a complete solution. Adaptive adversaries can find inputs that the model incorrectly classifies as legitimate instructions rather than override attempts. The boundary between a legitimate instruction and an injection attempt is inherently fuzzy, and the model's classification will have both false positives (refusing legitimate requests) and false negatives (following injected instructions).

The hierarchy is most effective when combined with structural defenses: input/output validation, tool-use restrictions, and sandboxed execution. Together, these layers create a defense-in-depth posture where each layer compensates for the weaknesses of the others. The instruction hierarchy reduces the frequency of successful injections, while structural constraints limit the damage when an injection succeeds.

Updated April 2025. Some URLs may require authentication for full access.