How to Manage LLM Context Windows for AI Agents
Context window exhaustion is the silent killer of production agents. You build a planning system that works beautifully in demos, then it runs for twenty tool calls and suddenly you're truncating critical state or blowing past your 128k token limit mid-execution. The problem isn't just size—it's that naive approaches to context management create cascading failures that are hard to debug.
Most teams start by stuffing everything into context: full conversation history, every tool call and response, all intermediate reasoning steps. This works until it doesn't. An agent debugging a production incident might execute fifty tool calls, each returning kilobytes of logs or trace data. At 4 tokens per word, you're burning through context fast. Worse, LLM performance degrades with bloated context even when you're under the limit—attention mechanisms struggle with irrelevant information, and latency scales with context size.
Middle truncation is the first strategy worth trying. Keep the initial prompt and system instructions, keep the most recent exchanges, drop everything in between. This preserves task definition and immediate context while shedding historical bloat. The failure mode is obvious: if the agent needs information from that middle section—a tool result from ten steps ago, a user clarification from earlier in the conversation—it's gone. You're trading completeness for predictability. For linear tasks where recent context dominates, this works. For agents that need to reference earlier work or maintain long-term state, it breaks down fast.
Deduplication and pruning help but only at the margins. If your agent calls the same API endpoint multiple times, you don't need five copies of identical error messages. If a tool returns a 50k character JSON blob but the agent only needs three fields, prune it before adding to context. These are hygiene fixes, not architectural solutions. You're buying yourself a 20-30% reduction in token usage, which delays the problem but doesn't solve it.
The more robust approach is retrieval-augmented memory. Instead of keeping everything in context, store tool outputs, conversation history, and intermediate state in a vector database or structured store. When the agent needs information, it queries the memory system and pulls in only what's relevant. This shifts the problem from "what fits in context" to "what's retrievable," which is a much easier problem to solve. The tradeoff is complexity—you need embedding models, retrieval logic, and careful prompt engineering to make the agent actually use the memory system. You also introduce retrieval failures as a new error mode. If your embedding model doesn't surface the right context, the agent proceeds with incomplete information and you won't know until it fails.
Sub-agent decomposition is the nuclear option. If a subtask generates massive amounts of context—say, analyzing hundreds of log files—spin up a dedicated sub-agent with its own context window. It does the work, summarizes the results, and returns a compressed output to the parent agent. This works but adds orchestration overhead and makes debugging significantly harder. You're now tracing execution across multiple agent instances, each with its own context and decision-making process.
The practical reality is you need a combination. Middle truncation for baseline hygiene, pruning for high-volume tool outputs, retrieval-augmented memory for tasks requiring long-term state, and sub-agents for genuinely unbounded subtasks. The key is understanding your agent's access patterns. Does it reference old context frequently or rarely? Are tool outputs large and structured or small and varied? Is the task inherently sequential or does it require random access to prior work? Your context strategy should match the task structure, not the other way around.