Context window

The context window is the maximum amount of text — measured in tokens — that an LLM can process in a single call. It covers everything: the system prompt, the conversation history, any supplied documents and the generated answer. Whatever does not fit into the window does not exist for the model.

Technically, the window size results from the model’s architecture and training. The attention mechanism relates every token to every other token, so the computational cost grows with length — long contexts are slower and more expensive. Windows have grown from a few thousand tokens to hundreds of thousands or even millions. A nominally large window does not mean uniform use of it, though: models can be worse at taking into account information buried in the middle of long contexts (“lost in the middle”).

In practice, the context window determines how much knowledge can be supplied per request — central to RAG and to AI agents, whose history and tool outputs gradually fill the window. A common misconception is that an LLM “remembers” conversations: in reality, the history is resent with every call; persistent memory only arises through external storage. Costs hinge on it, too — usage is billed by tokens, and needlessly long contexts cost money and answer quality.