Quick Answer
Use compaction for long, iterative work and retrieval for large or changing knowledge, then combine both when an agent must run reliably for extended sessions. Keep operational thresholds well below an LLM's hard limit because attention and recall can degrade before the window is full.
Introduction
A larger context window is not a memory strategy. Long-running agents accumulate instructions, tool schemas, tool outputs, retrieved files, and prior decisions until relevant facts become difficult to surface reliably. This is the practical problem often described as context rot: more tokens can mean less usable signal. Teams need an architecture that preserves decisions without carrying every intermediate artifact forward.
Key Takeaways:
Compaction preserves a working narrative while removing expensive conversational history.
Retrieval loads evidence only when the current task needs it.
Reliable agents separate durable memory from temporary execution context.

Context Window Management for Long-Running Agents
Context window management is an engineering discipline, not a prompt-writing trick. A large language model context window holds the active material a model can consider during one request, but relevance competes with volume as that material grows. Current research on context-window control shows why selective recall matters: ARC reported 99.40% exact-answer accuracy on a Needle-in-a-Haystack evaluation, versus 88.12% for its best baseline; on the harder LongBench-v2 subset, that same sliding-window baseline dropped to 30.67%.[1]
What fills the active context
Most agents exceed useful context capacity because each loop appends material that was necessary once but is irrelevant now. Tool-heavy workflows are especially vulnerable because raw outputs can dwarf the user request that triggered them.
System prompt: Defines operating rules, roles, and safety boundaries.
Tool definitions: Add schemas, parameter descriptions, and usage constraints.
Retrieved documents: Supply evidence for the immediate task.
Message history: Carries decisions, corrections, and unresolved work.
Tool results: Often contain verbose logs, payloads, and duplicate data.
Why bigger windows still fail
LLM context window limits are not merely a request-rejection boundary. As history expands, an agent can overweight recent tool output, miss an earlier constraint, or treat a stale observation as current. That creates a business problem as well as a technical one: longer calls raise token consumption, latency, and the cost of diagnosing failures after deployment.
For security-sensitive workflows, excess context also expands attacker-influenced material. Research summarized in agent red-teaming guidance found task-hijacking success rose from 11% to 81% with techniques tailored to LLM-backed agent behavior; across 25 repeated attempts, average attack success rose from 57% to 80%.[2]
Compaction vs Retrieval: The Operational Tradeoff
Compaction reduces what the agent carries. Retrieval decides what the agent should load. Mature agent architectures use both, but they assign each technique a distinct responsibility.
Compaction preserves the thread
Compaction summarizes completed work near a chosen token threshold, starts a fresh turn from that summary, and drops raw history that no longer needs to be model-visible. The lightest version clears old tool results after extracting facts needed for the next action. Microsoft's compaction guidance documents token-triggered compaction and its built-in 4-character-per-token estimator; use a model-specific tokenizer for accurate counts.[3]
Choose compaction when continuity matters more than source-level precision. A coding agent may preserve accepted decisions, current files, failed approaches, and pending tests, while discarding terminal chatter. This is a practical approach to reducing token overhead, but summaries can omit edge cases, exact wording, or provenance.
Compaction should optimize for recall before polish. Keep a structured record such as: "Decision: use schema version B. Constraint: do not alter public endpoint behavior. Pending: reproduce timeout." Providers document server-side approaches differently, so verify current provider documentation before relying on automatic behavior.
Retrieval restores the evidence
Retrieval keeps source material outside the active prompt and fetches only the passages needed for the current decision. A small, stable corpus can be pre-retrieved and pinned; a large or changing corpus needs search and read tools that load relevant passages just in time. Good RAG pipelines returns lightweight references first, then expands only the evidence an agent elects to inspect.
Treating long context windows and RAG as alternatives is therefore the wrong binary. Long context supports active reasoning over a bounded working set, while retrieval supplies grounded material on demand. Retrieval quality depends on chunking, metadata, ranking, and source freshness, so a weak index can return plausible but wrong context.
Task shape | Primary technique | What stays active | Main risk |
|---|---|---|---|
Long iterative coding | Compaction | Decisions, constraints, pending tests | Lost implementation detail |
Changing knowledge base | Retrieval | Task instructions and selected passages | Stale or irrelevant results |
Parallel research | Sub-agents plus retrieval | Condensed findings and citations | Conflicting summaries |
Tool-heavy workflow | Compaction plus pruning | Current state and recent outputs | Context drift |
Set compaction and pruning triggers below the hard model boundary so the system retains room for tool calls, error recovery, and the final response. The exact threshold depends on model encoding, tool payload size, and the amount of output the next operation requires.

Build a Context Control Plane, Not a Bigger Prompt
Managing context efficiently starts with explicit state categories: durable facts, task plans, temporary observations, and raw artifacts. Store each category differently, then require agents to justify why a record belongs in the active window. This avoids treating conversation history as a database.
Use external notes and clean sub-agent handoffs
Structured notes outside the prompt should hold stable decisions, source references, ownership, and unresolved questions. A specialist sub-agent should begin with a clean window, retrieve its own evidence, and return a condensed result rather than its full scratchpad. This reduces coordination noise, though it introduces a verification step when summaries disagree.
Use vector databases for semantic discovery, but retain identifiers, timestamps, and access controls alongside embeddings. Search alone is not enough when a result must be tied back to a current source or an authorized user scope.
Cache stable instructions, prune volatile artifacts
Prompt caching can reduce repeated processing of stable system instructions and tool schemas, but it does not solve relevance. Prune verbose logs, duplicate tool responses, and superseded plans before every handoff. Strong API design for AI agents also keeps tool responses narrow, paginated, and machine-readable so agents do not spend their usable context decoding presentation-oriented payloads.
Common failures are predictable: a stale note becomes assumed truth, a compacted summary drops a critical exception, or a chain of agents inherits far more history than its next task needs. Treat summaries as artifacts that require versioning and validation, not as authoritative memory.
Build vs Managed Agent Infrastructure
Building context infrastructure internally means owning orchestration, memory schemas, retrieval quality, token budgets, tracing, authorization boundaries, and incident analysis. That control can be necessary for proprietary workflows, but it also makes context behavior a permanent platform responsibility. Teams evaluating vector database options should separate retrieval storage decisions from the larger question of who operates the agent lifecycle.
What a managed layer changes
A managed platform can centralize execution environments, tracing, and operational controls, but teams still need to define what an agent may retain, retrieve, and pass onward. DevvPro frames this as an engineering ownership decision: managed operations do not remove the need for clear memory contracts and measurable failure handling.
Agent swarms from Grep's sandboxed agent execution run managed AI agents in which a lead agent assembles specialists that divide work, share context, and check results in isolated sandboxes with an audit trail.[4]

Conclusion
Use compaction to preserve an evolving workstream without retaining every turn, and retrieval to bring external evidence into scope only when needed. Use structured notes and narrow handoffs to protect durable decisions from raw execution noise. For teams building long-running agents, the API, retrieval, and orchestration boundaries determine whether an agent remains debuggable at scale.
Need a sharper architecture for long-running agents? Explore DevvPro engineering guidance for practical design decisions.
Frequently Asked Questions (FAQs)
What is a context window in LLMs?
A context window in LLMs is the active set of instructions, conversation history, tool definitions, and retrieved material a model can consider during a request, and it is separate from durable storage that an application may maintain outside the model.
How do RAG systems bypass context window limits?
RAG systems bypass context window limits by storing source material externally and retrieving selected passages at request time, which lets an agent work from relevant evidence without placing an entire document collection into every prompt.
What happens when you exceed the LLM context window?
When you exceed the LLM context window, the request must be rejected, truncated, or compacted depending on the implementation, and truncation can silently remove instructions or historical details that the next decision depends on.
How do you manage memory in long-context AI applications?
You manage memory in long-context AI applications by separating durable notes, retrievable evidence, current task state, and disposable tool artifacts, then defining explicit rules for what enters, leaves, and persists beyond the active prompt.
Why do models lose information in large context windows?
Models lose information in large context windows because relevant facts compete with a growing volume of unrelated or conflicting tokens, which makes correct retrieval and attention less reliable even when the request remains below the model's maximum capacity.
How do you implement effective context pruning techniques?
You implement effective context pruning techniques by deleting raw outputs after extracting needed facts, replacing resolved discussion with structured decisions, retaining current constraints, and validating that every retained item has a defined use in the next agent action.
Sources
- Addressable Recall Compaction for Long Context-Window Control in AI Agents — arxiv.org ↩
- NIST AI Agent Security: Red-Teaming Guidance and Enterprise Compliance — Lab Space ↩
- Compaction — learn.microsoft.com ↩
- Grep's sandboxed agent execution — grep.ai ↩


