What Is a Context Window? — The AI’s Working Memory Explained

Every large language model has a limit on how much text it can process at one time. This limit is called the context window, and it is measured in tokens. Everything the model considers when generating a response — the system instructions, the conversation history, the documents provided, and the user’s question — must fit within this window. Anything outside it is invisible to the model.

What the context window contains

The context window is not just the user’s latest message. It holds everything the model is currently working with: the system prompt that sets the model’s behaviour and role, the full conversation history up to the current turn, any documents or retrieved chunks provided as context, and the current question or instruction. All of this competes for the same limited space.

Why context length matters in practice

Short context windows mean the model can only consider a limited amount of information at once. In a long conversation, once the exchange exceeds the context window, the earliest messages start falling out. The model appears to ‘forget’ things mentioned earlier — because they are no longer in the window it can see. For document processing, a short context window means only a small portion of a long document can be fed to the model at a time. This is one of the core reasons RAG was developed: rather than feeding an entire document library into the context, only the most relevant chunks are retrieved and placed in the context window where they can influence the response.

How context lengths have grown

Early GPT models had context windows of around 4,000 tokens — roughly 3,000 words. Modern models have dramatically larger windows. Claude’s context window supports up to 200,000 tokens, which is roughly 150,000 words or a full-length novel. GPT-4 supports 128,000 tokens in its largest configuration. These larger windows make it possible to feed entire documents, lengthy reports, or extended conversation histories into the model’s context.

Longer context does not mean unlimited

Even with very large context windows, there are practical limits. More tokens in the context window means more computation — which means slower and more expensive inference. Research has also shown that models tend to pay more attention to information at the beginning and end of the context window, sometimes missing important information buried in the middle — a phenomenon called the lost-in-the-middle problem. For large-scale document retrieval, RAG remains more efficient and reliable than stuffing everything into a single large context.

The simple rule

The context window is the AI’s working memory for a single interaction. Everything it can see is in the window. Everything outside it does not exist for that interaction. Understanding this explains most of the surprising behaviours people encounter when using AI on long documents or extended conversations.

🎁 Try MarineRef free for 1 year — use code FREE2026 at maritime.rangalabs.cloud

👉 Get one AI tip every day on WhatsApp — free. Join here: https://chat.whatsapp.com/DhaGgTuQ9GE67ykGVMgxXb