1. What Is a Context Window?

In the first guide, we established that a language model predicts the next token based on the tokens that came before it. But the model cannot look back infinitely. There is a ceiling on how many tokens it can consider at once, and that ceiling is called the context window.

The best way to think of it is as the model's working memory. When you paste a long document into a chat, the model reads it into its context window. When you have a multi-turn conversation, each earlier message stays in the window so the model can reference what was said. The context window is the total space available for everything the model needs to hold in mind to produce its next response.

A model with a 4,000-token context window can hold roughly 3,000 words of combined input and output. A model with a 128,000-token window can hold about 96,000 words, enough for a short book. The difference in what you can do with those two models is enormous. The 4,000-token model chokes on a research paper. The 128,000-token model can read several papers and compare them in a single response.

2. What Fills the Context Window

People often assume the context window is the size of their question. In practice, multiple things consume tokens before the model generates a single word of response.

The practical implication is that a model with a 128,000-token context window does not let you paste 128,000 tokens of input. After the system prompt, conversation history, and a safety margin for the model's response, you might have 120,000 tokens of usable input space. In a multi-turn conversation, each turn eats into that budget, and eventually the oldest messages need to be dropped or summarised to make room for new ones.

3. Why There Is a Limit

The context window is finite because of how self-attention works. As we covered in the first guide, self-attention lets every token in the input look at every other token to determine relevance. This is what gives Transformers their power, but it comes with a computational cost that grows quadratically.

Quadratic growth means that doubling the number of tokens in the context window does not double the compute. It quadruples it. A model processing 1,000 tokens computes 1 million attention relationships. At 2,000 tokens, that becomes 4 million. At 10,000 tokens, 100 million. This is why early models had small context windows. The hardware could not handle the computation that larger windows demanded.

Recent years have seen dramatic improvements. Techniques like rotary position embedding (RoPE), sparse attention, and flash attention have reduced the compute penalty of long contexts. Hardware has gotten faster. These advances are why a model in 2022 had a 4,000-token window and a comparable model in 2025 can handle 128,000 or even a million tokens. The fundamental quadratic scaling has not gone away, but it has been mitigated enough to make large context windows practical.

4. The Lost-in-the-Middle Problem

A larger context window lets you put more text in, but it does not guarantee the model uses all of that text equally. Research published in 2023 by Liu et al. found that language models pay uneven attention across long inputs. Information at the beginning and end of the context gets more weight than information in the middle. The phenomenon is called the lost-in-the-middle effect, and it has practical consequences for anyone building LLM applications.

If you paste a 50-page document into a prompt and ask the model to find a specific detail, the model is more likely to find it if it appears on the first or last page. If the detail is on page 27, the model may miss it entirely, even though the document is well within the context window. The information is there. The model just does not attend to it as strongly.

This finding has direct implications for prompt design and RAG systems. If you are retrieving documents and placing them in the context, put the most relevant ones at the beginning and end of the context block, not in the middle. If you are writing instructions, put the most important ones at the top of the prompt and repeat them just before the model starts generating. The lost-in-the-middle effect is also an argument for keeping context lean. Fewer tokens mean the model's attention is more concentrated, and the probability of missing something important goes down.

5. Cost and Latency Trade-offs

Larger context windows cost more money and take more time. Both effects are linear in token count for most API pricing models, but the latency effect also has a quadratic component because of the attention computation.

On the cost side, API providers charge per token for both input and output. A prompt with 50,000 tokens of retrieved documents costs roughly 50 times more per call than a prompt with 1,000 tokens. For an application making thousands of calls per day, the difference between a 1,000-token prompt and a 50,000-token prompt can be the difference between a manageable bill and an eye-watering one. This is why RAG systems are designed to retrieve only the relevant chunks rather than dumping the entire knowledge base into the context.

On the latency side, the model takes longer to produce its first token when the context is large. This is called time-to-first-token, and it matters for user experience. A chatbot that takes 8 seconds to start responding because it is processing a 100,000-token context feels broken. For interactive applications, keeping the context lean is also a user experience decision.

The relationship between context size and latency is not strictly linear. The initial processing of the input, called prefill, scales with the number of tokens. But once the model starts generating output, each new token only needs to attend to the already-processed context, which is faster. So the first token is slow and subsequent tokens are faster. The longer the context, the longer the wait for that first token.

6. Context Window Sizes Across Models

Context windows have grown dramatically since ChatGPT launched in late 2022 with a 4,000-token limit. The table below shows how far the field has come and where the major models stand as of mid-2026.

The jump from 4,000 to 2 million tokens in under three years represents a 500x increase. Whether this trend continues depends on whether the computational challenges of quadratic attention can be solved at scale, or whether alternative architectures replace the Transformer entirely. For now, the practical question is not whether you have enough context, but whether you are using the context you have wisely.

7. Managing Context Limits

Even with large context windows, you need strategies for managing what goes into them. The goal is to maximise the signal the model receives while minimising the cost and latency of processing it.

8. Security Implications of Long Context

A larger context window gives the model more text to work with, but it also gives an attacker more surface area to exploit. Research from Anthropic demonstrated that models with longer context windows are more vulnerable to jailbreaking attacks, where adversarial text buried deep in the context attempts to override the model's safety instructions.

The mechanism is straightforward. In a long context, the model's attention is spread across many tokens. Safety instructions in the system prompt get diluted when thousands of tokens of user-provided content follow them. An attacker can embed instructions in a retrieved document, a pasted article, or a long conversation history that steer the model toward producing harmful output. The larger the context, the more room there is to hide such instructions.

Mitigations include keeping contexts as small as possible, using delimiters to separate instructions from data (as covered in the prompt engineering guide), treating all user-provided content as untrusted, and monitoring model output for signs of successful injection. No single mitigation is sufficient. Defense in depth, combining prompt-level controls with output filtering and tool-level permissions, is necessary for any system processing untrusted input.

9. Practical Guidelines

Based on everything covered so far, here are practical recommendations for working with context windows in production.

10. The Future of Context Windows

Context windows have grown 500x in three years, and there is active research on pushing them further. Google's Gemini 1.5 Pro already handles 2 million tokens, and experimental models have demonstrated effective attention over even longer sequences. Whether this growth continues depends on whether the quadratic scaling of attention can be circumvented at scale.

Several research directions are promising. Sparse attention mechanisms skip irrelevant token pairs entirely, reducing the compute from quadratic to near-linear for long sequences. Alternative architectures like Mamba and state space models replace the attention mechanism with a fixed-size recurrent state, which does not grow with sequence length at all. If these architectures prove as capable as Transformers for language tasks, the context window constraint may eventually disappear.

For now, the context window remains a practical constraint that shapes how LLM applications are built. The strategies in this guide, from RAG to summarisation to careful prompt ordering, exist because the window is finite. Understanding the constraint and designing around it is a core skill for anyone building with language models.

Frequently Asked Questions