How LLMs Are Trained: Pre-training, Example Answers and RLHF

What you'll learn#

• Explain the three stages that turn a next-word guesser into a chat assistant: pre-training, example answers, and RLHF.
• Say why a model that has only read text continues your message instead of answering it.
• Walk through how people's rankings become a scorer, and how the scorer shapes the assistant.
• Spot what training can't do: teach your company's rules on the spot, or make the model care about truth.
• Choose between a base model and a chat model, and know which stages an engineer really touches at work.

What to know first: what an LLM is (*What is an LLM?*), that it reads and writes in tokens (*Tokens and tokenization*), and that it writes one token at a time by guessing the next one (*Next-token prediction*). A neural network is the big web of adjustable numbers inside the model (*AI, ML, deep learning and LLMs*). That's all you need.

The problem: who taught it to answer?#

Aaryan's cousin is getting married on Friday, so Aaryan needs the day off from Kelvin, their manager. They open a chatbot and type: "Write my boss a leave request for Friday." A polite letter appears in seconds: "Dear Kelvin, may I take Friday off for my cousin's wedding? My work is covered. Thanks, Aaryan." Ready to send.

Here's the puzzle. In the last few lessons you saw that an LLM does one thing: it guesses the next token, again and again. Nothing in "guess the next word" says *answer the person*, *be polite*, or *get the day right*. So where did those habits come from?

The answer is training, and it happens in stages. Think of how a child learns to talk. First they hear years of talk at home and pick up the language. Then grown-ups show them how to reply: "When someone asks, you answer." Then they get smiles for the good replies and frowns for the rude ones. An LLM goes through something similar, in three stages. Where the picture breaks: a child lives in the world and understands it; the model only ever sees text and learns patterns in it, and most of its "smiles" come from a scorer, not from a person each time.

The big picture: three stages#

Each stage starts from the model the stage before produced. The first stage is enormous and costly; the last two are small by comparison. The table below shows how different their sizes are, using the numbers labs have published.

Stage 1: pre-training, reading to guess the next word#

Pre-training is the model reading a huge pile of text and practising one game on it: hide the next word, guess it, check. The text comes from web pages, books, code and more. The model starts out knowing nothing; its numbers, called parameters (the adjustable numbers inside the neural network), start out random.

1. Take a sentence and hide the next word. "I'll be on leave on ___."
2. Guess. The model gives a chance to every possible next token. Early on, its guesses are poor.
3. Check and nudge. The real word was *Friday*. A measure called the loss says how surprised the model was by the real word. The parameters get nudged a tiny bit so that *Friday* becomes a little more likely next time.

Repeat that over trillions of tokens and the model picks up grammar, spelling, facts that appear often, how code is laid out, and how letters, recipes and essays usually go. Meta reported that Llama 3 was pre-trained on over 15 trillion tokens, using up to 16,000 GPUs at once. Nobody labels this text by hand; the next word in the text is its own answer key. That's why it scales so far.

Why reading alone isn't enough: the base model#

The model you get after pre-training is called a base model: a very good continuer of text, and nothing more. Ask it for something, and it treats your message as the start of a document and writes what usually comes next on a web page like that.

Notice the third row. If you start the letter yourself, the base model finishes it, because that's what text on a page does. It has the skill to write the letter; it doesn't have the habit of answering. You might think: let it read even more. That makes a better continuer of web pages, not an assistant. The fix has to come from a different kind of lesson.

Stage 2: example answers teach the shape of a reply#

In stage two, people write example conversations: an ask, then the reply a good assistant would give. The model trains on these the same way as before, guessing the next token, but now every text it sees has the shape *someone asks, the assistant answers*. This stage is called supervised fine-tuning, or SFT. Fine-tuning means further training a model that's already trained, on a smaller set of chosen examples; you'll learn when to do it yourself in *When to fine-tune*.

Thirteen thousand examples sounds tiny next to trillions of tokens, and it is. That's the point: the language and knowledge were already there from pre-training. Stage two only teaches the *format*: when a message is an ask, the next thing is an answer. To mark who is speaking, the conversation is wrapped in special tokens, a chat template, so the model can tell the user's turn from its own. Models released after this stage often carry "Instruct" or "Chat" in their name.

Stage two has a limit. The model learns to imitate the examples, but it can only be as good as the replies people wrote, and writing thousands of great replies is slow and costly. Ask the stage-two model the same thing twice and you may get two quite different letters, some good, some not. Which one should it prefer?

Stage 3: RLHF, learning from people's picks#

Judging is easier than writing. You may not write the perfect leave letter, but you can tell which of two is better. RLHF, reinforcement learning from human feedback, uses exactly that. Reinforcement learning means learning by trying things and getting marks for them, the way a game player improves from scores. The idea of learning from people's preferences goes back to a 2017 paper by Christiano and colleagues.

1. Drafts. The stage-two model writes several replies to the same ask.
2. People rank them. A person orders the drafts from best to worst. In InstructGPT, labellers ranked between 4 and 9 drafts per prompt, for about 33,000 prompts.
3. Train a scorer. The rankings train a second model, the reward model, that reads a reply and gives it a score: a guess at how much people would like it.
4. Practise against the scorer. The assistant writes replies to fresh asks, the reward model scores each one, and the assistant's parameters are nudged toward higher scores. A leash keeps it close to the stage-two model, so it can't drift into odd text that tricks the scorer.

Why a scorer instead of people marking every reply? Because the practice step needs millions of marks, and a reward model gives them in milliseconds. People's time goes into the rankings once; the scorer stretches it. The payoff can be large: in the InstructGPT study, people preferred replies from a 1.3-billion-parameter model trained this way over the 175-billion-parameter GPT-3 base model, over a hundred times bigger.

Today there are cheaper cousins of this stage. DPO (direct preference optimisation) learns from the same pairs of better and worse replies, but skips the separate scorer and the practice loop. People often call the whole family *preference tuning*. The idea stays the same: people's picks decide which replies the model makes more often.

A code sketch: guessing and scoring in miniature#

Real training adjusts billions of parameters on GPUs. This toy shows the two core ideas with plain Python: stage one counts which word follows which, and stage three learns a score for each draft from people's pairwise picks.

import math
from collections import Counter, defaultdict

# Stage 1, pre-training in miniature: count which word follows which
text = "i need friday off . i need friday off for a wedding . i need monday off ."
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    follows[a][b] += 1
print("after 'need':", follows["need"].most_common())

# Stage 3, a scorer learned from people's picks (winner first)
picks = [("short and clear", "long and stiff"), ("short and clear", "wrong day"),
         ("long and stiff", "wrong day"), ("short and clear", "long and stiff")]
score = {"short and clear": 0.0, "long and stiff": 0.0, "wrong day": 0.0}
for _ in range(200):
    for win, lose in picks:
        p_win = 1 / (1 + math.exp(score[lose] - score[win]))  # chance the scorer agrees
        score[win] += 0.1 * (1 - p_win)    # nudge the winner up
        score[lose] -= 0.1 * (1 - p_win)   # and the loser down
for draft, s in sorted(score.items(), key=lambda kv: -kv[1]):
    print(f"{draft:16} {s:+.1f}")

• Lines 1–2 bring in the maths module and two counting helpers.
• Lines 5–9 are stage one in miniature: split a tiny text into words, then for every pair of neighbours count how often the second follows the first.
• Line 10 asks what follows *need*: *friday* twice, *monday* once. That count is a crude next-word guess; a real model learns these chances with a neural network instead of a table.
• Lines 13–15 are four rankings from people, winner first, and a starting score of zero for each draft.
• Line 18 works out how likely the current scores say the winner is to win (the same formula real reward models are trained with).
• Lines 19–20 nudge the winner's score up and the loser's down, more when the scorer was surprised.
• Lines 21–22 print the learned scores: the short, clear draft ends highest and the wrong-day draft lowest, the same order people picked.

Common misconceptions#

• "The chatbot learns from me while we talk." Training is finished before you chat; the model you're talking to doesn't change from your messages. A company may use chats, depending on your settings, to train a *later* version, but not the one in front of you. Within one chat it only "remembers" what's in its context window.
• "Reading more would make it a good assistant." More pre-training makes a better continuer of text. Answering, format and manners come from the later, much smaller stages.
• "RLHF teaches it the truth." RLHF teaches what people *preferred*. People can't check every fact, and they often prefer answers that agree with them or sound confident. Research has found RLHF-trained assistants sometimes agree with a user's wrong belief for this reason. Making things up confidently is called hallucination, the topic of a later lesson.
• "A person rates every answer it gives." People rank a fixed set of drafts once. After that, a reward model does the marking, millions of times.
• "Training stores the internet so it can look things up." It stores no pages to search. It holds patterns in its parameters, which is why it can be wrong about details it saw only a few times.

In real work#

Almost no AI engineer pre-trains a model. A pre-training run for a large model costs millions of dollars in GPU time, so it's done by a handful of labs. What you do instead is choose, steer and sometimes adapt the result.

• Pick the right release. Many labs publish both a base model and an instruct or chat model. For a chat feature you want the instruct one; base models are for research, for text completion, or as a starting point for your own fine-tuning. An open-weight model (one whose parameters you can download, covered in a later lesson) often comes in both versions.
• Use the chat template. If you run a model yourself, wrap messages in the template it was trained with. Skip it and an instruct model can start behaving like a base model again.
• Don't train for facts or rules. For Aaryan's company leave policy, hand the model the policy with the question; that's quicker, cheaper and easy to update. Training changes behaviour and style better than it adds fresh facts.
• Collect feedback. The thumbs up and down buttons in chat apps gather the same kind of preference data RLHF uses. If you fine-tune later, those pairs are valuable.
• Know the cost of preference tuning. Full RLHF needs ranked data, a reward model and a delicate training loop. DPO and similar methods are what most small teams reach for when they do preference tuning at all.

Recap#

1. Pre-training: the model reads trillions of tokens, guessing each next word and getting nudged. The result, a base model, knows language but continues text instead of answering.
2. Example answers (supervised fine-tuning): people write thousands of ask-and-reply pairs; the model learns the shape of a reply.
3. RLHF: people rank drafts, a reward model learns their picks, and the assistant practises toward higher scores. It learns which reply people prefer, not what's true.

Practice#

1. Recall: In your own words, why does a base model continue "Write my boss a leave request" with more requests?
2. Recall: What does the reward model take in, and what does it give back?
3. Recall: Why do labs train a scorer instead of having people mark every practice reply?
4. Challenge: In the code sketch, add a fourth ranking where *wrong day* beats *long and stiff*, run it again, and explain how the scores change and why.

Practice

Check your understanding

FAQ

Frequently asked questions

Sources

Where the facts come from

• Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., 2022)
• The Llama 3 Herd of Models (Meta, 2024)
• Deep reinforcement learning from human preferences (Christiano et al., 2017)
• Direct Preference Optimization (Rafailov et al., 2023)
• Towards Understanding Sycophancy in Language Models (Sharma et al., 2023)