What Are Tokens in AI? Why Some Models Miscount the R's
An LLM never reads letters or whole words. It reads tokens: pieces of text, each swapped for a number. Here's how that works, and why it decides limits, prices and spelling mistakes.
The strawberry quiz
Strawberry has two r's. That's what the chat assistant said. Meera, who's ten, counted three on their fingers. Earlier that Sunday evening, Arjun had asked the assistant to quiz Meera for their spelling homework: how many r's in strawberry? A ten-year-old just beat the computer at spelling.
The same assistant writes lovely sentences, so why can't it count letters? This lesson answers one question: what does an AI model actually read when you type? By the end you'll know what a token is, why rare words cost more of them, and why length limits, prices and letter counting all come back to tokens. (The story is an illustration of a well-known failure, not a test of one named model today.)
Think of a rubber-stamp printing set
In the last lesson we met large language models (LLMs): models that learned to write from lots of text. To see what they read, picture a child's rubber-stamp printing set.
- Big stamps for common whole words, like
theorschool. - Small stamps for word pieces, like
st,rawandberry. Line them up and you can printstrawberry. - A number on every handle, so each stamp can be named by its number.
Now imagine someone who only ever sees the handle numbers. They can print any word, but they never see a single letter. That someone is the model.
How text becomes tokens, in three steps
- Cut the text into pieces. Common words stay whole. Rare words are split into smaller pieces: a word part, sometimes a single letter. On its own,
strawberrybecomes three pieces:st(one piece),rawandberry. - Swap each piece for its number. The numbers come from a fixed list of pieces, learned from lots of common text: pieces that show up often earned their own entry.
st·raw·berrybecomes 302 · 1618 · 19772. The model only ever receives these numbers, never the letters. - Count everything in pieces. The reply comes back as pieces too, joined into text for you. How much a model can read at once, how much it can write, and the price are all counted in these pieces.
- Your text (strawberry) to Cut into pieces (tokenizer)
- Cut into pieces (tokenizer) to Piece-number list (vocabulary)
- Piece-number list (vocabulary) to The model (sees 302 · 1618 · 19772)
Each piece is a token. Cutting text into tokens is tokenization. The tool that does it, with its fixed list (its *vocabulary*), is the tokenizer. OpenAI's open-source tokenizer is called tiktoken, and the counts in this lesson come from o200k_base, the name of one of OpenAI's vocabularies. Other models can use other vocabularies.
Meera's spelling list, in tokens
Meera's list this week has four words: the, school, strawberry and chrysanthemum. Here it is run through tiktoken with o200k_base. Each word is counted on its own, with nothing before it.
| Word | Letters | Tokens | Pieces |
|---|---|---|---|
| the | 3 | 1 | the |
| school | 6 | 1 | school |
| strawberry | 10 | 3 | st · raw · berry |
| chrysanthemum | 13 | 4 | ch · rys · anthem · um |
the
- Letters
- 3
- Tokens
- 1
- Pieces
the
school
- Letters
- 6
- Tokens
- 1
- Pieces
school
strawberry
- Letters
- 10
- Tokens
- 3
- Pieces
st·raw·berry
chrysanthemum
- Letters
- 13
- Tokens
- 4
- Pieces
ch·rys·anthem·um
o200k_base vocabulary, each word on its own.Common words are one token however many letters they have. Rarer words cost more pieces. anthem gets its own piece because it's a common word on its own. Now the twist: inside a sentence, a word usually has a space in front of it, and the space is part of the piece. strawberry with its space is a single token (number 101830), and chrysanthemum with its space is three ( chrys · anthem · um).
So Meera's question, How many r's in strawberry?, became seven tokens: How · many · r · 's · in · strawberry · ?. The model got only the number for strawberry, not the letters inside it. The lone r in the question was its own token, but the model had no way to look inside strawberry and check its r's. So, in this example, it answered from patterns it had picked up during its training, and got it wrong.
The catch: a token is not a word
Most people think a token is a word, so a hundred words would be a hundred tokens. But a token is a piece of text. Common words are one token; rare words, names, long numbers and many non-English scripts often take several. The tiktoken README gives a rule of thumb: for English, a token is about four letters on average. That's an average, not a promise for any one word.
- The same word can be one token or several.
strawberryon its own is three;strawberryin a sentence is one. - Different tokenizers count differently. Two models can give the same message different token counts.
- Messages often cost more tokens than words. Plan with token counts, not word counts.
Why tokens matter when you build
- Limits. A model can only take in and write a certain number of tokens. The most it can read at once is called its *context window*: the amount of text it can hold in view at one time, which has its own lesson.
- Prices. Companies charge by the token, often quoted per million tokens; providers such as Anthropic price usage this way. So a long chat costs more than a short one.
- Speed. A model writes its answer one token at a time, so longer answers take longer.
- Letters. The model never sees letters directly, so spelling, letter counting and exact character edits are unreliable.
Quick check: which costs more tokens, the school (two words, nine letters) or chrysanthemum (one word, thirteen letters)? chrysanthemum: it's rare, so on its own it splits into four pieces, while the school is two tokens, one per common word.
Next, we'll see what the model does with those numbers: how an LLM predicts the next token, which is the idea called *next-token prediction*, where the model picks the most likely next piece, one piece at a time.
Summary
Key takeaways
- An LLM reads tokens, not letters or words: common words are one token, rare ones split, and every limit and price is counted in tokens.
- Tokenization cuts text into pieces and swaps each piece for a number from the tokenizer's fixed vocabulary.
- The same word can be one token or several:
strawberryin a sentence is 1 token,strawberryon its own is 3 (o200k_base). - For English, a token is about four letters on average, but rare words, names and many non-English scripts cost more.
- Because the model never sees letters, spelling and letter counting are unreliable.
Interview corner
What is tokenization, and why should an engineer building on an LLM care?
Tokenization cuts text into tokens, whole common words or pieces of rare ones, and maps each to a number from the tokenizer's fixed vocabulary; the model only ever sees those numbers. It matters because limits, prices and speed are all measured in tokens, so a message can cost more than its word count suggests, and because the model never sees letters, spelling and letter counting are unreliable.
Practice
Check your understanding
0 of 5 answered
Finished this lesson?
Mark it done to track your progress. Saved in this browser only.
FAQ
Frequently asked questions
What is a token in AI?
A token is a piece of text that a language model reads as one unit, swapped for a number. Common words are usually one token; rare words are split into several pieces.
How many words is 1,000 tokens?
There's no fixed ratio. The tiktoken README puts the average at about four characters of English per token, so English prose usually has fewer words than tokens. Rare words, code and many non-English scripts use more tokens.
Why can some AI models miscount the r's in strawberry?
Many current models get it right, but the classic slip comes from tokens: the model never sees the letters. With OpenAI's o200k_base tokenizer, strawberry inside a sentence is a single token, one number with no letters inside it, so a model can answer from learned patterns instead of counting.
Do all models count tokens the same way?
No. Different models can use different tokenizers and vocabulary, so the same text can give different token counts. Use the tokenizer that belongs to the model you call.
Sources
