Context Window Explained: Why an AI Model Forgets
An AI model can only see a fixed amount of text at once: its context window. When your text doesn't fit, part of it is cut, and the model seems to forget.
The night before the exam
It's the night before Arjun's economics exam. They paste a whole semester of notes, six chapters, into an AI study helper. Then they ask a question from Chapter 1: "What is opportunity cost?"
The helper replies: "Your notes don't cover opportunity cost." But they do. It's the first page of Chapter 1. Arjun can see it in the file. So why can't the helper?
Because the model never read Chapter 1. It only reads what fits in its context window, and Chapter 1 didn't make it in.
What a context window is
Think of a whiteboard of a fixed size. The model can only read what is written on it right now. That whiteboard is the context window: the amount of text a model can see at once.
Its size is counted in tokens, the small pieces a model splits text into. A short word like "cost" is usually one token; a long word may be two or three. As a rough rule of thumb, one token is about three quarters of an English word, so 100 tokens is about 75 words.
Everything shares that one board. In Arjun's case, four things fill it:
- The instructions: the app's own setup text, which tells the model how to behave.
- The pasted text: all six chapters of Arjun's notes.
- The question: "What is opportunity cost?" In a longer chat, every earlier message goes here too.
- The answer: the reply being written takes room on the same board.
Why the model forgets
The model has no memory of its own. Each time Arjun asks something, the app sends everything again: the instructions, the notes and the question. The model reads it fresh, writes an answer and keeps nothing. So if something isn't on the board, the model simply doesn't know it.
- Arjun's question (input) to Adds all six chapters (pasted text)
- Adds all six chapters (pasted text) to Fits in the window? (context window)
- Fits in the window? (context window) to Model writes answer (output)
- Fits in the window? (context window) to Chapter 1 cut (truncation)
Let's count. In this example, assume the helper's window is 10,000 tokens (real ones are much bigger, but the maths is the same).
| On the board | Tokens |
|---|---|
| Instructions | 400 |
| Chapter 1 notes | 1,500 |
| Chapters 2 to 6 notes | 8,000 |
| Arjun's question | 100 |
| Room for the answer | 1,500 |
| Total | 11,500: 1,500 over the 10,000 limit |
Instructions
- Tokens
- 400
Chapter 1 notes
- Tokens
- 1,500
Chapters 2 to 6 notes
- Tokens
- 8,000
Arjun's question
- Tokens
- 100
Room for the answer
- Tokens
- 1,500
Total
- Tokens
- 11,500: 1,500 over the 10,000 limit
This app cut the notes from the start until they fit, so Chapter 1 went. Apps differ here: many refuse with a "too long" error, some summarise, and some search the file and send only the parts that look relevant. Whatever the app does, the model only answers from what reaches the board.
Bigger windows, and Meera's fix
Windows vary a lot between models; many current ones offer from about 128,000 up to a million tokens or more. Using the rule of thumb above, a million tokens is roughly 750,000 words, about 1,500 pages at 500 words a page. That's approximate, but it shows the scale.
Bigger is not free, though. A fuller window means a slower answer and a higher bill, because you pay per token. And models can miss details buried in the middle of a very long input, even when it all fits.
Arjun's classmate Meera does it differently. They paste only Chapter 1 and ask the same question. That's 400 + 1,500 + 100 + 1,500 = 3,500 tokens, well inside the window, and the answer is right.
- Paste only the part you need, one chapter at a time.
- Summarise a long chat and carry the summary into a new one.
- Start fresh for a new task, and repeat the key facts.
One sentence to remember: the model only sees what fits in its window right now, so keep what matters in it.
Summary
Key takeaways
- The context window is how much text a model can see at once, counted in tokens.
- A token is a piece of a word; as a rough rule, about three quarters of an English word.
- The window holds everything: instructions, pasted text, the chat so far and the answer.
- The model has no memory; the app re-sends everything each time, and what doesn't fit is cut or refused.
- Bigger windows are slower and cost more, and details in the middle can be missed.
- The model only sees what fits in its window right now, so keep what matters in it.
Practice
Check your understanding
0 of 3 answered
Finished this lesson?
Mark it done to track your progress. Saved in this browser only.
FAQ
Frequently asked questions
What is a context window in an AI model?
It is the amount of text the model can read at once, counted in tokens. It holds the instructions, the conversation so far, anything you paste in and the answer being written.
Why does an AI forget the start of a long document or chat?
The model keeps nothing between requests. The app sends everything again each time, and when it doesn't fit in the window, some apps cut the oldest part, others summarise it or refuse with an error. Whatever isn't sent, the model never sees.
How big is a 1 million token context window?
Roughly 750,000 English words, about 1,500 pages at 500 words a page. It's an estimate, because the number of tokens per word depends on the text and the model.
