Next-Token Prediction: How an LLM Writes One Token at a Time
An AI chat never has a finished answer waiting. It scores every possible next token, picks one, adds it to the text and runs again, until it picks a stop token.
The notice that stopped mid-sentence
It's Sunday morning in a housing society. Arjun is on the residents' committee, and he asks a chat assistant to write a notice for the lift-lobby notice board: the water will be off on Sunday, from 10 to 2. The reply starts to appear word by word, as if someone is typing it.
Meera, a neighbour waiting to print it, says what most people think: "It's already written. It's just showing off slowly." So Arjun taps Stop. The notice ends right there: "Dear residents, the water supply will be off on". Nothing more ever arrives. There was no rest of the notice hiding anywhere.
So how does an AI chat write its answer, and why does it appear word by word? By the end of this lesson you'll be able to walk through the exact loop that writes every reply, and say why Stop leaves half a sentence.
A worksheet where the blanks keep coming
Think back to a school fill-in-the-blank: "The sun rises in the ____." You wrote "east" without being told, because you had read that pattern many times. You didn't plan the rest of the page. You just filled the one blank in front of you.
Now picture a strange worksheet. Each time you fill a blank, a new blank appears straight after your word. You fill that one too, and another appears, again and again, until you write "the end". That is very close to how a large language model writes.
One thing from the last lesson: models don't read words, they read tokens, small pieces of text that are often a whole word or part of one. In this lesson, most of our tokens happen to be whole words, so "word" and "token" line up.
The loop: score, pick, add, repeat
Every reply is written by the same three steps, run over and over.
- Score. The model reads all the text so far: Arjun's request plus "Dear residents, the water supply will be". It gives every token in its vocabulary (the full list of tokens it knows, tens of thousands, or even hundreds of thousands) a score for how well it fits next.
- Turn scores into chances, then pick one. The scores become chances that add up to 100%. The model picks one token: usually the likeliest, sometimes not. In this example it picks " off".
- Add it and run again. The picked token goes on the end of the text, and the whole loop runs again on the slightly longer text. It keeps going until it picks a special stop token, a token that means "I'm done", or until it reaches a length limit.
- Text so far (the prompt + reply so far) to Model scores tokens (LLM)
- Model scores tokens (LLM) to Chances add to 100% (probabilities)
- Chances add to 100% (probabilities) to Pick one, add it (next token)
That is called next-token prediction: predicting the one token that comes next, given all the text before it, again and again. Researchers also call a model that writes this way autoregressive: each new piece it writes is fed back in as input for the next one.
Worked example: Arjun's notice, run by run
Here is Arjun's notice being written. The chances below are made up for this example, not taken from a real model, but the shape is real: a few strong candidates, then a long tail of everything else.
| Run | Text so far ends with | Top chances (example) | Picked |
|---|---|---|---|
| 1 | …the water supply will be | off 62% · stopped 18% · cut 11% · other 9% | off |
| 2 | …will be off | on 71% · from 14% · for 9% · other 6% | on |
| 3 | …will be off on | Sunday 80% · Saturday 6% · this 5% · other 9% | Sunday |
| 4 | …off on Sunday | from 48% · between 30% · . 12% · other 10% | from |
| Last | …from 10 am to 2 pm. | [stop] 55% · Please 35% · other 10% | [stop] |
1
- Text so far ends with
- …the water supply will be
- Top chances (example)
- off 62% · stopped 18% · cut 11% · other 9%
- Picked
- off
2
- Text so far ends with
- …will be off
- Top chances (example)
- on 71% · from 14% · for 9% · other 6%
- Picked
- on
3
- Text so far ends with
- …will be off on
- Top chances (example)
- Sunday 80% · Saturday 6% · this 5% · other 9%
- Picked
- Sunday
4
- Text so far ends with
- …off on Sunday
- Top chances (example)
- from 48% · between 30% · . 12% · other 10%
- Picked
- from
Last
- Text so far ends with
- …from 10 am to 2 pm.
- Top chances (example)
- [stop] 55% · Please 35% · other 10%
- Picked
- [stop]
Each row is one full run of the model, so four new words took four runs. Notice run 4: "from" wins with 48%, but "between" was close behind with 30%. The model is less sure there, and a different pick would have taken the notice a different way. In the last row the stop token wins, and the notice is finished.
Now Arjun's Stop button makes sense. He tapped it after run 2, so the loop ended and the text stopped at "…will be off on". Run 3, the one that would have added "Sunday", never happened (the red row). Stop doesn't hide a finished answer. It ends the loop, and the words that were never picked never exist.
The common mistake: "it's already written"
Most people think, like Meera, that the AI writes the whole answer first and then plays it back slowly for effect. It doesn't. Each token is only decided when it appears. The word-by-word display is the model actually working.
- That's why Stop leaves half a sentence: there is no rest of the answer to show.
- That's why a model can't go back and change a word it already wrote: every earlier token is now part of the text it reads.
- That's why longer answers take longer: every new token is one more run of the model.
Interview corner and what to remember
If an interviewer asks: "How does an LLM generate a response?" A strong answer: it generates one token at a time. At each step it scores every token in its vocabulary given all the text so far, turns those scores into probabilities, picks one, appends it, and runs again on the longer text. It stops when it picks an end-of-sequence token or hits a length limit, so the whole answer is a loop of next-token predictions.
The one sentence to keep: An LLM writes one token at a time: score every option, pick one, add it to the text, and run again. Next, we'll see why that growing text can't grow forever, and why a long AI chat starts to forget.
Summary
Key takeaways
- An LLM writes one token at a time: score every option, pick one, add it to the text, and run again.
- At each step the model scores every token in its vocabulary, and the scores become chances that add up to 100%.
- It usually picks the likeliest token, but not always.
- The loop ends when the model picks a special stop token or reaches a length limit.
- Nothing is written ahead: tapping Stop ends the loop, which is why the reply ends mid-sentence.
Interview corner
How does an LLM generate a response?
It generates one token at a time. At each step it scores every token in its vocabulary given all the text so far, turns those scores into probabilities, picks one, appends it, and runs again on the longer text. It stops when it picks an end-of-sequence token or hits a length limit, so the whole answer is a loop of next-token predictions.
Practice
Check your understanding
0 of 5 answered
Finished this lesson?
Mark it done to track your progress. Saved in this browser only.
FAQ
Frequently asked questions
Why does ChatGPT type word by word?
Because that is how it writes. The model produces one token, adds it to the text, and runs again to produce the next. The app shows each token as soon as it's picked, so you see the answer being written, not a finished answer played back.
Does an LLM plan the whole answer before writing it?
No. Each token is decided only when it is generated, based on all the text before it. Nothing after the current token exists yet, which is why stopping a reply leaves it mid-sentence.
How does an LLM know when to stop?
It stops when it picks a special end-of-sequence token, a token that means the reply is done, or when it reaches a length limit set by the app.
Sources
