What Is a Context Window? LLM Memory, Explained

A context window is the maximum amount of text a large language model can hold in view at one time — everything it reads and everything it writes, counted together. It’s the model’s working memory. If the conversation, documents, and instructions you feed a model exceed that limit, the oldest or least relevant material has to be dropped, because the model physically cannot attend to text outside the window.

Measured in tokens, not words

Context windows are counted in tokens, not words or characters. A token is a chunk of text — often a word, a word fragment, or a piece of punctuation. As a rough rule of thumb, one token is about four characters of English, so 1,000 tokens is roughly 750 words. The exact split depends on the model’s tokenizer, and code, non-English text, and unusual formatting tokenize less efficiently.

The crucial point is that the window covers both directions. If a model has an 8,000-token window and your prompt uses 6,000 tokens, only about 2,000 tokens remain for the response. Run out of room and the reply gets cut off. This is why long documents and long back-and-forth conversations eventually hit a wall — you’re spending the same budget on input and output.

To understand why the limit exists, it helps to know how these models work. As covered in our explainer on what an LLM is, the model processes the entire window at once through the attention mechanism at the heart of the transformer architecture. Attention compares every token against every other token, so the compute cost grows roughly with the square of the window length. Doubling the context can quadruple the work — which is why large windows are expensive to serve and why they were, for years, small.

What fills the window

In a typical application, the context window holds several things stacked together:

• The system prompt — standing instructions that define the model’s role and rules.
• The conversation history — every prior user message and model reply in a chat.
• Retrieved documents or data — files, search results, or database rows pulled in to ground the answer.
• The current user message — the immediate question or task.
• Room for the response — the tokens the model still needs to generate its reply.

All of it competes for the same fixed budget. A long system prompt and a long history leave less room for the actual question and answer.

What happens when you run out

Models don’t gracefully “remember” text that falls outside the window — it’s simply gone. Applications handle this in a few ways. The bluntest is truncation: drop the oldest messages until the rest fits. More sophisticated approaches summarize earlier turns into a compact recap that costs fewer tokens, or use retrieval to pull only the relevant passages back in on demand.

That last technique is the foundation of retrieval-augmented generation, which pairs a model with a search step so it can answer questions about far more text than would ever fit in the window. Instead of stuffing an entire knowledge base into the prompt, RAG fetches the few passages that matter — often using vector embeddings to find them by meaning — and drops just those into the context.

Bigger windows aren’t a free lunch

Modern models advertise very large context windows, and it’s tempting to treat that as unlimited memory. Two caveats matter.

First, cost and latency scale with usage. You pay per token, and processing a huge prompt is slower and more expensive. Sending 100,000 tokens of context on every request adds up fast. Techniques like prompt caching help by reusing the processed form of a stable prefix — like a long system prompt — across requests, so you don’t pay to re-read the same tokens every time.

Second, using the window well is a skill. Research and practice both show that models attend most reliably to material at the very start and very end of a long context, and can overlook details buried in the middle — the so-called “lost in the middle” effect. A window that technically fits 100,000 tokens doesn’t guarantee the model weighs all of them equally. Putting the most important instructions and data near the edges, and keeping the prompt focused, produces better answers than dumping everything in and hoping.

Context window vs training knowledge

A common confusion is mixing up the context window with what the model “knows.” They’re separate:

	Context window	Training knowledge
What it holds	Text from the current request	Patterns learned during training
When it’s set	At inference, per request	Baked in when the model was trained
How it changes	You control it every prompt	Fixed until retrained or fine-tuned
Freshness	Whatever you paste in	Frozen at the training cutoff

The window is how you give a model fresh or private information it never saw in training — the day’s data, your internal docs, the current conversation. The training knowledge is the general capability underneath. Good applications combine the two: a capable base model, fed the right context at the right time.

The takeaway

A context window is an LLM’s fixed working memory, measured in tokens, shared between the prompt and the response. Everything the model considers has to fit inside it, and the attention math means bigger windows cost more than proportionally. Don’t treat a large window as infinite memory — manage it. Keep prompts focused, summarize or retrieve instead of pasting everything, put the important material at the edges, and cache stable prefixes. Working within the window deliberately beats trying to fill it.