Context rot is the measurable decline in LLM output quality as input context grows longer, even when the task stays exactly the same, and even when the model is nowhere near its context window limit. Chroma's 2025 technical report tested 18 frontier models and found that every single one degraded as input length increased.

A 1M-token context window doesn’t mean you can safely use 1M tokens. This article covers what the benchmarks actually show, why context degradation happens, and the context engineering techniques that mitigate it.

What Is Context Rot?

Context rot (also called context degradation) happens when an LLM's reliability decreases as the number of input tokens increases, holding task difficulty constant. The model still produces fluent output, but it produces worse output: missed facts, confident hallucinations, ignored instructions, or outright refusals.

Context rot is not context window overflow. Overflow is a hard failure. You exceed the token limit and the request errors out or truncates. Rot happens well inside the limit, which makes it far more dangerous: nothing visibly breaks, so teams don't notice until users complain about accuracy.

The Benchmark Evidence for Context Rot

Most long-context benchmarks accidentally conflate input length with task difficulty (longer inputs usually mean harder problems). Chroma's methodology held task complexity constant and varied only input length, which lets you attribute the performance drop to length alone.

Four experiments stand out.

Needle in a Haystack, Made Realistic

The standard Needle in a Haystack (NIAH) test, where models retrieve a planted sentence from a long document, shows near-perfect scores for frontier models, which is why providers cite it. But NIAH tests lexical matching: the question uses the same words as the needle.

Chroma extended the test to semantic matches, where the model must connect a question to an answer phrased differently. Performance dropped as context grew, and low-similarity needle-question pairs degraded fastest. Real-world queries are almost always semantic, not lexical, which means the benchmark your vendor quotes is the easy version of the problem you actually have.

Distractors Amplify the Damage

Adding distractors, content that is topically related to the question but doesn't answer it, made degradation significantly worse, and the effect compounded with length. Model families also failed differently. Claude models tended to abstain when uncertain, while GPT models were more prone to hallucinating a confident wrong answer.

Retrieved chunks that are almost relevant are precisely what a mediocre retrieval step produces. Sloppy retrieval doesn't just waste tokens; it actively injects the distractor pattern that degrades answers the most.

The Structure Finding

Chroma's strangest result was that models performed better on shuffled haystacks than on logically coherent documents, which was a consistent pattern across all 18 models. Notably, Chroma doesn't claim to know why. The report suggests input structure may influence how attention is applied and explicitly flags the mechanism as an open question for interpretability research.

Conversational Memory: LongMemEval

The most production-relevant experiment used LongMemEval, a benchmark simulating chat assistants that must answer questions using long conversation histories. Models were tested in two conditions: the full input (the original LongMemEval prompt averaging ~113,000 tokens, mostly irrelevant chat history) and a focused version averaging just ~300 tokens, containing only the parts relevant to the question.

Every model performed substantially better in the focused condition. Same question, same relevant facts, same model. The only difference was the irrelevant context surrounding them. This is the direct, quantified case for retrieval over stuffing: forcing a model to find the signal and reason about it in one step costs real accuracy.

Even at maximum simplicity, the effect held. In a task that asked models to merely replicate a sequence of repeated words, performance still degraded with length. With bizarre failure modes, from Claude Opus 4 and GPT-4.1 occasionally refusing outright (Opus 4 sometimes citing copyright concerns) to Gemini models producing essentially random output.

Lost in the Middle

Context rot describes what happens as length grows. The lost in the middle effect, describes what happens depending on where information sits.

In the foundational study, Liu et al. (TACL 2024) tested multi-document question answering and key-value retrieval while moving the relevant document through different positions in the context. The result was a characteristic U-shaped curve. Accuracy was highest when the answer appeared at the beginning or end of context, and dropped significantly when it sat in the middle. No model they tested used all positions equally well.

Follow-up work confirms this isn't a solved problem. Princeton's HELMET benchmark extended the analysis to 128K-token inputs and found models still prefer recent context and struggle to recall facts from the middle and early portions. And research on positional attention bias points to a mechanism: models allocate disproportionate attention to early and late tokens regardless of where the relevant content actually is.

In a long prompt, where you place critical instructions and facts changes whether the model uses them. The middle of your context is the worst real estate you own.

Why Context Rot Happens

There's no single agreed-upon cause, but three factors show up consistently in the research.

Attention is a finite budget. In a transformer, every token can attend to every other token, n² pairwise relationships for n tokens. As Anthropic's engineering team puts it, models have a limited "attention budget" that gets stretched thinner as context grows. More tokens means less attention per token, and the relationships that matter get diluted by the ones that don't.

Training distributions favor short contexts. Models see vastly more short sequences than long ones during training, so the positional patterns they learn are strongest at lengths they've practiced. Long-context capability is partly extrapolation and extrapolation is where reliability goes to die.

Interference scales with content. Every additional passage is another candidate for the model to confuse with the answer. Chroma's distractor experiment showed this directly: even a single near-miss passage reduced accuracy versus a clean baseline, and four distractors compounded the damage further.

Research from Du et al. found that performance degrades even when models perfectly retrieve the relevant information and the drop persists even when all irrelevant tokens are replaced with whitespace or masked entirely. Sheer input length alone impairs reasoning, independent of retrieval quality and with nothing to be distracted by.

How Context Engineering Mitigates Context Rot

If long context degrades output, the answer isn't shorter tasks. It's deliberately controlling what enters the window. This discipline is context engineering, and it has largely replaced prompt engineering as the core skill for building reliable LLM systems. Three techniques do most of the work.

1. Compaction: Summarize, Don't Accumulate

Compaction takes a conversation or agent trajectory approaching problematic length, distills it into a high-fidelity summary, and continues in a fresh window seeded with that summary. It's the first lever Anthropic recommends for long-horizon coherence, and it's what tools like Claude Code do automatically when sessions run long.

Compaction preserves central facts and decisions but loses verbatim detail. The skill is deciding what your agent must retain (task state, key findings, open questions) and writing the compaction prompt to protect exactly that.

2. Retrieval: Fetch Only What the Task Needs

The LongMemEval result is the argument for retrieval in one sentence: models given only relevant context significantly and consistently outperformed models given everything. Instead of loading full histories, full documents, or full schemas, retrieve the slice that answers the current question and present that.

This is also why "RAG is dead, just use the big window" aged badly. Retrieval isn't a workaround for small context windows; it's a defense against context degradation in large ones. Fewer, more precisely selected passages beat a larger volume of loosely relevant ones and they're cheaper and faster too.

3. Structured Context: Give the Model a Map

Raw dumps rot fastest. Structured context (clear sections, distilled notes, explicit schemas, external memory files the agent writes to and reads from) keeps signal density high. Anthropic's structured note-taking pattern has agents persist state outside the window and pull it back only when needed; LangChain documents similar patterns across production agent stacks, including isolating sub-tasks in separate context windows via subagents.

For data and analytics use cases specifically, the highest-leverage structure is a semantic layer: a curated definition of your metrics, dimensions, and relationships. An analytics agent reasoning over a semantic layer works with a few thousand high-signal tokens instead of raw schema dumps and query logs, exactly the "smallest set of high-signal tokens" principle applied to data.

What This Looks Like in Practice

• Trigger compaction on thresholds, not at the window limit; degradation starts early, so compact while quality is still high.
  
  

• Rank and prune retrieved content aggressively. Near-relevant chunks are distractors, and distractors are the worst thing you can add.
  
  

• Put critical instructions at the start or end of context, never the middle; work with the U-curve, not against it.
  
  

• Evaluate at your real context lengths. A prompt that works at 5K tokens is a different system at 80K. Test the version you actually ship.

Frequently Asked Questions

What is context rot in LLMs? 

Context rot is the decline in an LLM's output quality as input context grows longer, even on tasks of constant difficulty. The term comes from Chroma's 2025 research showing all 18 tested frontier models degrade non-uniformly with input length.

Is context rot the same as "lost in the middle"? 

No, but they're related. Lost in the middle is a positional effect, where models recall information at the start and end of context better than the middle. Context rot is a length effect, where quality degrades as total input grows. In production, you typically suffer both at once.

At what context length does degradation start? 

There's no universal threshold. Degradation is continuous and varies by model, task, and content. Du et al. measured substantial performance drops within the first several thousand tokens on reasoning tasks, while simple lexical retrieval holds up far longer. The safe assumption: if your context is long enough that you're wondering, it's long enough to test.

Does a bigger context window fix context rot? 

No. Window size determines when you hit overflow, not how well the model uses what's inside. Chroma's results show degradation well within advertised limits across every model tested, which is why context engineering, not window size, is the mitigation.

Is RAG still necessary with million-token context windows? 

Yes, arguably more so. The LongMemEval experiments showed every model performed significantly better on ~300-token focused inputs than on the same questions buried in ~113K tokens of mostly irrelevant history. Retrieval is how you produce that focused context at scale.

How do I detect context rot in my own application? 

Evaluate the same task at multiple context lengths and compare accuracy. Because rot is silent (outputs stay fluent while quality drops) logging and evals are the only reliable detection method. Watch especially for accuracy dropping as conversation or session length grows.

Sources

1. Chroma Research: Context Rot: How Increasing Input Tokens Impacts LLM Performance (Hong, Troynikov, Huber, July 2025). The technical report that coined the term; 18-model evaluation. https://research.trychroma.com/context-rot

2. Liu et al., TACL 2024: Lost in the Middle: How Language Models Use Long Contexts. Foundational study on positional degradation. https://aclanthology.org/2024.tacl-1.9/

3. Anthropic Engineering: Effective Context Engineering for AI Agents. Attention budget framing; compaction, note-taking, and subagent techniques. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

4. Yen et al. (Princeton): HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. Extends lost-in-the-middle analysis to 128K tokens. https://arxiv.org/abs/2410.02694

5. Hsieh et al.: Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Mechanism behind positional degradation. https://arxiv.org/abs/2406.16008

6. Du et al.: Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. Shows length degrades reasoning even with perfect in-context retrieval. https://arxiv.org/abs/2510.05381

7. Wu et al.: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. The conversational benchmark used in Chroma's focused-vs-full experiments. https://arxiv.org/abs/2410.10813

8. LangChain: Context Engineering for Agents. Production patterns for retrieval, summarization, and context isolation. https://blog.langchain.com/context-engineering-for-agents/

9. Kamradt: Needle in a Haystack: LLM Retrieval Test. The original benchmark Chroma's experiments extend. https://github.com/gkamradt/needle-in-a-haystack