Context Window

What is a context window, and how does it differ from the "memory" discussed elsewhere?

A context window refers to the upper limit on how much text a model can process at once when generating a single response, measured in tokens — the basic unit a model processes text in, roughly corresponding to about 0.75 English words per token. The system prompt, everything in the current conversation turn, any retrieved documents, and tool description text all count toward this same total limit. Once the sum exceeds this limit, the model can't see everything at once — the earliest portions get pushed out of the window.

This is an entirely different layer from long-term memory: long-term memory is a storage mechanism specifically designed to persist information across conversations, typically living in a database or vector index, retrieved and written into the context window only when needed. The context window itself is simply the capacity limit on "how much text sits in front of the model this time it's generating a response" — it isn't a memory system at all, it's more like the model's "desk size" while it works. No matter how big the desk is, anything that spills over the edge still gets pushed somewhere out of sight.

Why does this context window limit exist, and what drives it?

The core reason is that the internal computational mechanism models use to process text (self-attention) sees its computational cost rise sharply as input length increases — the model needs to judge how every piece of text relates to every other piece, so the longer the input, the more combinations need comparing, and the more computational resources get consumed. This means the context window isn't an arbitrary limit vendors impose deliberately — it's a technical boundary jointly determined by computational cost and hardware capability. Context window capacity has expanded significantly in recent years precisely because model architecture and hardware compute have kept advancing, gradually making computational costs feasible that used to be prohibitive.

Another driver comes from actual needs within agent scenarios themselves: if an agent needs to simultaneously read a long document, remember multiple turns of conversation history, and load descriptions from several tools to decide which one to use, all of this content stacked together can easily exceed earlier models' capacity limits. This is also why, as agent applications increasingly depend on processing large amounts of input simultaneously, expanding context window capacity has become one of the industry's ongoing areas of investment.

How does the context window actually affect an agent's real-world performance, and what details are worth watching for?

The first important detail is the gap between "advertised capacity" and "effective capacity": the number a vendor publishes for their context window represents the technical upper limit the model can accept as input, but independent testing broadly finds that a model's actual retrieval and reasoning quality degrades when handling input close to that limit — content buried in the middle of the input, neither at the beginning nor the end, is especially prone to being overlooked or misjudged. This means "how large this model's window is" and "how well this model performs when the window is full" are two questions that need to be evaluated separately, not just read off a vendor's published number.

The second important detail is that, in agent scenarios, the context window fills up much faster than in a simple chat application — every tool call, every document read, every message exchanged with another agent continuously consumes the same window allowance. If a task an agent needs to complete involves several steps in a row, the complete record of earlier calls can keep accumulating, approaching or even exceeding the window limit; once it's exceeded, the system typically needs to summarize, discard older content, or move some information out of the window into long-term memory retrieved only when needed, to keep the task able to continue. The third detail is that output limit and input window are two different things: even if a model can read in a large amount of text at once, the length of a single generated response usually has a separate cap. For coding-agent tasks that need to produce large amounts of content at once (editing multiple files, say), the output limit sometimes becomes the bottleneck earlier than the input window does, requiring the task to be split across multiple rounds to complete.

What does the context window mean for me, and how do I evaluate and use it practically?

If you're designing or using an agent system, understanding the context window's limitation helps you avoid a common misjudgment: assuming "the window is big enough" equals "you can just dump everything in and let the model sort it out." The more practical approach is to proactively filter and prioritize which content genuinely needs to sit in the current window — move infrequently changing information that can be summarized into long-term memory, retrieved as needed, and only put content directly relevant to the immediate step into the window, rather than unconditionally stuffing in everything available and hoping the model picks out what matters from the noise.

Another practical angle worth noting is that when comparing different models or different plans, don't just compare advertised window-capacity numbers — confirm how retrieval quality actually holds up under your specific usage scenario (how long the input typically is, where key information tends to sit within it). Two models that both advertise large-capacity windows may show a meaningfully different actual reliability gap on your specific task, and that gap typically only becomes clear through testing against your own scenario, not simply by comparing the numbers on a spec sheet.