Why AI Agents Fail: Context Engineering 101

by

Key Takeaways

Context engineering is the practice of designing everything an AI model sees at inference — not just the prompt, but its tools, memory, and retrieved data. The same model gives wildly different answers depending on what fills its context window and how. Anthropic frames it as managing a finite ‘attention budget,’ while Drew Breunig maps the four ways context breaks down. This guide walks through the core idea and the Write-Select-Compress-Isolate playbook from scratch.

  • Transformers compute n² relationships for n tokens, so the “attention budget” drains fast (Anthropic)
  • Four failure modes break context: poisoning, distraction, confusion, and clash (Drew Breunig)
  • Four strategies fix it: Write, Select, Compress, and Isolate (LangChain)
  • The goal is the smallest set of high-signal tokens — not a context window stuffed to the brim

Ask an AI chatbot the same question twice and you may get a sharp answer once and nonsense the next. The model didn’t change between the two tries. What changed is what sat inside its context window at that moment — and shaping that is the job of context engineering. This is the discipline of deciding what an AI agent knows before it acts, and in 2026 it has quietly become the skill that separates agents that work from agents that flail.

Through 2025, a phrase kept circulating in engineering circles: prompt engineering is dead, long live context engineering. That’s an overstatement, but the direction is right. Teams building autonomous agents learned the hard way that polishing a clever prompt does almost nothing once the model is juggling tools, documents, and a sprawling history.

This guide explains context engineering for AI agents from the ground up: what it is, why context degrades, and the concrete strategies practitioners use to fix it. You don’t need to write code to follow along — by the end, you’ll understand why your company’s chatbot keeps hallucinating, and where to look first.

AI-generated illustration — a glowing translucent cube representing an AI context window with data streams flowing in from tool, memory and document panels, evoking context engineering
Context engineering is the work of deciding how information from tools, memory, and documents gets loaded into a single context window — AI-generated illustration (OpenAI gpt-image-1-mini)

Why Prompt Engineering Hit a Wall

A few years ago, the marquee AI skill was prompt engineering. Phrase the question well and the answer improved dramatically. Adding a single line like “think step by step” could lift performance on its own.

That worked because the model saw only a tiny slice of text: your instruction. The trouble began when AI stopped answering one-off questions and started operating as an agent. An agent plans multiple steps, calls tools, and pulls in prior turns, documents, and search results as it goes. The payload feeding the model is no longer a one-line query — it’s a sprawling bundle of tens or even hundreds of thousands of tokens.

Andrej Karpathy captured the shift when he described context engineering as “the delicate art and science of filling the context window with just the right information for the next step.” The center of gravity moved from the question to the composition of the window itself. Prompting is still part of that composition, but it is now one layer inside a much larger problem.

So is prompt engineering dead? Not at all. It survives as a component — the instruction layer — inside the broader task of assembling context. That is precisely why Karpathy and Anthropic both describe it as the superset that contains prompt engineering, rather than its replacement.

💡 Related Posts
AI Agent vs Chatbot vs LLM: The 2026 Complete Guide

What Context Engineering Is: The Attention Budget

Anthropic defines context engineering as curating and maintaining the optimal set of tokens a model sees during inference — system instructions, tools, external data, and message history all included. Where prompting asks “what should I tell the model,” context engineering asks “what does the model need to know to do this well.”

Why does that composition matter so much? Because of a hard constraint baked into the Transformer architecture. A Transformer computes relationships between every pair of tokens, which means n tokens create n² pairwise relationships. Double the tokens and you quadruple the work the model must spread its focus across.

Anthropic calls the consequence an “attention budget.” A model has a finite amount of attention to allocate on any given pass, and every token you push into the window spends a little more of it. The context window is not an infinite filing cabinet — it is a scarce resource that depletes with use.

Here is the counterintuitive part: a bigger context window is not automatically better. Anthropic points to “context rot” — the tendency for retrieval accuracy to decline as the window grows. It isn’t a sudden cliff. It’s a gradual erosion of precision the longer the context runs.

In simpler terms: The attention budget is like a conference-room whiteboard. It’s wide but not infinite, and if you cram every note onto it, the one that actually matters gets buried. The skill isn’t writing more — it’s keeping only what counts.
A close up of a computer circuit board
As tokens grow, the n² math behind attention eats the budget the model has to spend — Photo by Luke Jones on Unsplash

How Context Breaks: The Four Failure Modes

If context is a finite resource, how exactly does it get wasted or poisoned? Developer Drew Breunig sorted the ways long contexts fail into four categories: poisoning, distraction, confusion, and clash. Each one has a concrete, measured example behind it.

The first is context poisoning. A hallucination or error slips into the context and then gets referenced again and again on later steps, corrupting the model’s judgment. Breunig’s example is Gemini 2.5 playing Pokémon: a misread game state got etched into the agent’s “goals” section, and the agent began fixating on objectives that were impossible to achieve.

The second is context distraction. When the context grows too long, the model over-indexes on that history and neglects what it learned during training. Gemini 2.5 Pro supports a million-token window, yet in the Pokémon setup it began repeating past actions instead of forming new plans once it crossed roughly 100,000 tokens. A separate Databricks study found the accuracy of Llama 3.1 405b starting to slip at around 32,000 tokens.

The third is context confusion. Irrelevant material sits in the context and the model uses it anyway, dragging answer quality down. On the Berkeley Function-Calling Leaderboard, every model performed worse once it was handed more than one tool. In the GeoEngine benchmark, a quantized Llama 3.1 8b failed with 46 tools available — even though they fit inside a 16k window — but succeeded once the list was trimmed to 19.

The fourth is context clash. Conflicting information or tools pile up and the model loses the thread. A Microsoft and Salesforce study delivered the same information in “sharded” pieces across several messages instead of all at once. Performance dropped by an average of 39%, and OpenAI’s o3 fell from 98.1 to 64.1.

The common thread is uncomfortable but clarifying. In all four modes, adding more to the context actively hurt the model. What matters is not the volume of information but the signal-to-noise ratio — the same reason capable agents ace benchmarks yet crumble in the real world.

Failure modeWhat goes wrongMeasured example
PoisoningAn error lodges in context and is cited repeatedlyGemini 2.5 Pokémon — fixated on impossible goals
DistractionOver-indexing on a long history, ignoring trainingGemini 2.5 Pro slips past ~100k / Llama 405b at ~32k
ConfusionSuperfluous info and tools get used anywayTrimming 46→19 tools flipped failure to success (GeoEngine)
ClashConflicting information accumulatesSharded delivery: -39% average, o3 98.1→64.1
Key point: The four failure modes share one lesson. Context doesn’t improve because you added more; loaded wrong, it degrades performance by amounts you can measure.
💡 Related Posts
AI Agent Benchmarks Are Breaking in 2026
A group of blue and green balls on a black background
Every token connects to every other token — the more you add, the denser the web — Photo by Ecliptic Graphic on Unsplash

The Four Strategies: Write, Select, Compress, Isolate

Now the fix. LangChain organizes the field’s practical toolkit into four strategies: Write, Select, Compress, and Isolate. Most production agents run on some blend of the four.

Write means storing information outside the context window. Instead of cramming everything into the window and blowing the attention budget, an agent parks what it doesn’t need right now. Anthropic’s multi-agent researcher saves its plan to memory — a scratchpad — once context risks getting truncated past 200,000 tokens. The way ChatGPT, Cursor, and Windsurf remember you across sessions is the same idea.

Select means pulling only what’s needed back into the window. The familiar technique of RAG (retrieval-augmented generation) lives here. Tellingly, applying retrieval to the tool descriptions themselves — so the agent only sees relevant tools — improved tool-selection accuracy roughly threefold. When Claude Code loads a project’s CLAUDE.md file, that too is a form of Select.

Compress means keeping only the tokens a task requires by summarizing the rest. As a long conversation nears the window’s limit, older stretches get replaced with a summary. Claude Code’s auto-compact triggers exactly this once the window hits 95% of capacity.

Isolate means splitting one large context into several so the parts can’t contaminate each other. The canonical form is a multi-agent setup. In Anthropic’s multi-agent researcher, each sub-agent burns roughly 15 times more tokens than a single chat, yet delivers better results by exploring in parallel. Crucially, each sub-agent hands the parent only a 1,000-to-2,000-token summary rather than its entire trail.

StrategyOne-line definitionReal systems
WriteStore outside the windowScratchpads (past 200k tokens), ChatGPT/Cursor memory
SelectPull in only what’s neededRAG, tool selection (~3× accuracy), CLAUDE.md
CompressSummarize to save tokensClaude Code auto-compact (at 95%)
IsolateSplit across sub-agentsMulti-agent (15× tokens, returns 1–2k summaries)
pastel spheres on gradient background
Write, Select, Compress, and Isolate aren’t rivals — they’re a toolkit used together — Photo by Milad Fakurian on Unsplash

Principles for Building Good Context

Knowing the strategies is one thing; assembling context in practice is another. Anthropic offers a handful of working principles, and one idea runs through all of them: find the smallest set of high-signal tokens that maximizes the odds of the outcome you want.

First, write system prompts at “the right altitude.” A rigid wall of hardcoded rules fails, and so does vague, hand-wavy guidance. The target is language specific enough that a person reads it and knows what to do, yet open enough to leave room for judgment.

Second, keep the tool set minimal and non-overlapping. The confusion failure mode came straight from tool overload. The more tools with fuzzy, overlapping purposes an agent holds, the harder it is for the model to choose. A short, unambiguous tool list wins.

Third, use compaction and agentic notes for long jobs. When a session runs long, summarize aggressively — preserve the architectural decisions and drop the redundant output — and let the agent jot durable notes to an external file like NOTES.md, kept outside the window. That way a lengthy run doesn’t lose the thread.

Fourth, retrieve data just in time. Rather than front-loading every document into context, the agent holds lightweight identifiers and calls a tool to fetch the actual content only when it’s needed. It is the most direct way to conserve the attention budget.

These principles converge on a single mental model. Context is a budget, not a warehouse. The moment you start asking what to leave out rather than what to add, you’re doing context engineering.

Key point: The playbook reduces to right-altitude prompts, minimal non-overlapping tools, compaction plus notes, and just-in-time retrieval — all in service of spending the attention budget wisely.

Context Engineering vs RAG vs Prompt Engineering

By now the vocabulary can blur together. Prompt engineering, context engineering, and RAG are not competing alternatives fighting for the same job — they sit in concentric circles, each nested inside the next.

The innermost circle is prompt engineering — refining the instruction handed to the model, still a vital ingredient. The larger circle around it is context engineering, which designs the evidence, tools, memory, and retrieval that let that instruction actually work.

So where does RAG fit? RAG is one branch of the outer circle — the Select strategy, pulling relevant information into the window. It is a subset of the outer circle, not a rival to it. That circle also covers Write, Compress, and Isolate, none of which RAG touches.

Hold that map and the practical conversation gets sharper. “Should I refine the prompt” is an innermost-circle question; “which documents should I retrieve” is a Select question; “the history got too long, how do I trim it” is a Compress question. Placing a problem at the right layer is already half the solution.

ConceptScopeRelationship
Prompt engineeringThe instruction given to the modelInnermost — one element of context
RAGRetrieving and injecting external infoThe Select strategy within context engineering
Context engineeringThe full composition of tokens in the windowOuter circle — contains both

The real move is a shift in perspective: from asking better questions to designing the information environment. That is what context engineering is really about, and it links naturally to the other pillars of the agent era — from reasoning models to prompt injection security.

Frequently Asked Questions (FAQ)

Q1. What’s the difference between context engineering and prompt engineering?

Prompt engineering refines how you instruct the model. Context engineering is the larger task of designing all the information a model sees — system prompt, tools, memory, retrieved data — so that the instruction actually works. Prompt engineering is one layer inside it, not a competitor.

Q2. Isn’t a bigger context window always better?

No. Because Transformers scale computation with the square of the token count, the “attention budget” drains quickly, and “context rot” erodes retrieval accuracy as the window grows. Gemini 2.5 Pro supports a million tokens yet degraded past roughly 100,000 in testing. What matters is loading the smallest set of high-signal tokens, not filling the window.

Q3. Is RAG the same thing as context engineering?

RAG (retrieval-augmented generation) is one branch of context engineering. It maps to the “Select” strategy in LangChain’s framework — pulling relevant information into the context window. The discipline is broader, also covering the Write, Compress, and Isolate strategies.

Q4. What exactly is the attention budget?

A Transformer computes relationships between every pair of tokens, so n tokens create n² relationships. As tokens grow, that computation balloons and each token gets a shallower slice of the model’s attention. Anthropic calls this finite pool the “attention budget,” and spending it wisely is the heart of the discipline.

Q5. How does context compaction work?

When a long conversation’s history nears the window limit, older material is summarized down to its essentials and the rest is dropped. Claude Code’s auto-compact runs automatically once the window reaches 95% of capacity. The principle is to preserve important information like architectural decisions while discarding redundant output.

Q6. Do I need to be a developer to care about context engineering?

If you use or evaluate AI agents, it helps. The same model produces very different results depending on what information it’s given and in what order. Understanding the four failure modes — poisoning, distraction, confusion, clash — gives you the instinct to diagnose why a chatbot keeps going off the rails.

References

Stay Ahead of the Curve

Get the latest insights on AI, tech, and markets — delivered in both Korean and English.

Explore More Posts