Context windows are not memory

Every few months the context window gets bigger, and every few months a wave of posts declares retrieval dead. The reasoning goes: if the model can hold your entire codebase, why bother chunking, embedding, and ranking anything? Just put it all in the prompt.

The reasoning is wrong, but not for the reason people usually give. The problem isn't that big contexts are expensive or slow, though they are both. The problem is that a context window is not memory. It is a working set — everything the model can see on this one forward pass, reconstructed from scratch every single time you call the API.

The API is stateless, and that changes everything#

Here is the part that trips up people coming from ordinary application development. There is no session on the other end. When you send a twenty-turn conversation, you are not appending to something the server remembers — you are re-sending all twenty turns, and the model is reading them for the first time, again.

// Turn 5 doesn't "continue" turn 4. It re-sends turns 1-4 plus the new message.
const messages = [
  { role: "user", content: "Let's use Postgres for this." },
  { role: "assistant", content: "Good choice. Here's a schema..." },
  // ...16 more turns...
  { role: "user", content: "Wait, what database did we pick?" },
];
 
const response = await client.messages.create({
  model: "claude-opus-5",
  max_tokens: 16000,
  messages, // the entire history, re-read from zero
});

This has a consequence people underrate: anything not in that array does not exist. Not "is hard to recall" — does not exist. If your agent summarized a decision three turns ago and you dropped the summary to save tokens, the decision is gone. Not fuzzy. Gone.

It has a second consequence people underrate in the opposite direction: anything that is in that array gets attention, whether it deserves it or not. Stale tool output from twelve steps ago is competing for the model's attention with the instruction you wrote thirty seconds ago.

Why "just put everything in" degrades#

Attention is a finite resource being divided across whatever you hand over. Push in 400,000 tokens where 8,000 were relevant, and you have not given the model more information — you have given it a worse signal-to-noise ratio and asked it to do the retrieval itself, in a single forward pass, with no ability to backtrack.

The failure mode is specific and recognizable once you have seen it:

• The model answers using something plausible from the middle of a long document rather than the authoritative statement near the end.
• Instructions given early get quietly overridden by patterns established later.
• In agent loops, the model starts repeating work it already did, because the evidence that it did the work is buried under 60 tool results.

None of these look like "the model is bad at long context" on a benchmark. They look like your product being subtly unreliable in ways that are annoying to reproduce.

Four things people call "memory"#

Once you stop treating the context window as memory, you notice that "memory" is four unrelated problems wearing a trench coat.

1. Working set. What the model needs on this call. This is the context window, and its job is to be small, relevant, and ordered so the most important material is not buried.

2. Durable state. Facts that must survive the process: decisions made, constraints agreed, files written. This belongs in a database, a file, or a memory store — somewhere that outlives the conversation. The Claude API offers a memory tool for exactly this, and it works because it makes the storage explicit rather than hoping the transcript retains it.

3. Retrieval. The ability to go find durable state relevant to right now and promote it into the working set. This is what RAG actually is, stripped of the acronym: a function from "current situation" to "what should be in the window."

4. Compaction. What to do when the working set outgrows the window anyway. Summarize the old turns, keep the summary, drop the raw material.

Big context windows help with exactly one of these — they raise the ceiling on the working set. They do nothing for the other three. An agent with a 1M-token window and no durable state still forgets everything the moment the session ends.

Designing the working set on purpose#

The practical shift is to stop thinking of the message array as a log and start thinking of it as a document you are assembling for a specific reader with a specific question. Some rules that follow directly:

Put stable content first, volatile content last. This is partly about attention and entirely about caching. Prompt caching is a prefix match — the cache key comes from the exact bytes up to each breakpoint, so a timestamp interpolated into your system prompt invalidates everything after it. Stable system prompt, then tools, then history, then the new turn.

Don't edit the system prompt mid-conversation. It sits at the front of the prefix; changing it re-processes the entire cached history at full price. On current Claude models you can append a role: "system" message inside the messages array instead — the cached prefix survives, and it reads as an operator instruction rather than user text.

const response = await client.messages.create({
  model: "claude-opus-5",
  max_tokens: 16000,
  system: [
    { type: "text", text: STABLE_PROMPT, cache_control: { type: "ephemeral" } },
  ],
  messages: [
    ...history,
    { role: "user", content: userMessage },
    // Appended after the cached prefix — history stays cached.
    { role: "system", content: "Terse mode: keep responses under 40 words." },
  ],
});

Prune tool results aggressively. In a long agent loop, the majority of your tokens are tool output, and the majority of that is irrelevant within two steps. A file you read to check one function does not need to stay in context for the rest of the session. Context editing (clearing old tool results) and compaction (summarizing them) both exist for this; the choice is whether you need the gist later or nothing at all.

Write decisions down somewhere real. If a fact matters beyond this turn, it belongs in storage, not in the hope that the transcript survives compaction. A plain Markdown file the agent reads and writes is a completely legitimate implementation and beats an elaborate vector store for most single-agent workflows.

The uncomfortable part#

Doing this well means your prompt-assembly code becomes a real component with real logic — selection, ordering, budgeting, eviction. It is the least glamorous part of an LLM application and the part that most determines whether the thing works.

The good news is that it is ordinary engineering. Deciding what a function needs to see, in what order, within what budget, is a problem software engineers have been solving since forever. The only novelty is that the "function" charges you per token and gets worse at its job when you hand it too much.

Bigger context windows are genuinely useful. They mean fewer hard failures, more headroom for long tool loops, and less aggressive chunking. What they are not is permission to stop thinking about what goes in the window. The window is a budget, and someone has to spend it deliberately. Right now, that someone is you.