LLM Evals: Why "It Looks Good" Isn't Enough

What are LLM evals?#

LLM evals are systematic, repeatable tests that judge an LLM, or a system built on one such as an agent, against clear criteria. They're how you know an AI app works, instead of hoping it does.

It's the step most AI engineers skip. Building the feature feels like the work; checking it properly feels optional. It isn't. Three words in that definition do the heavy lifting:

• Systematic: a planned set of test cases, not whatever prompts come to mind.
• Repeatable: you re-run the same tests after any change to the model, the prompt or the workflow.
• Clear criteria: you define what a good answer looks like up front, before you read any answers.

Why manual testing fails#

Here's how many AI features ship: an engineer runs the app with four or five prompts, the answers look fine, and it goes to production. That feels like testing, but it leaves big gaps.

• Edge cases go unseen. Five prompts you wrote yourself cover the happy path, not the odd, vague or tricky things real users type.
• There's no baseline. Nothing is written down, so there's no way to compare before and after a change.
• Regressions hide. A prompt tweak that fixes one answer can quietly break others, and you'd never know.

Traditional software vs LLM apps#

Classic unit tests assume the code is deterministic: call a function with the same input and you get the same output, so one passing test is strong evidence. LLM apps break that assumption, so the way you test them has to change too.

• Behaviour: traditional software is deterministic; an LLM app is probabilistic.
• Output: traditional software gives the same output for the same input; an LLM app can give different outputs.
• Strategy: with traditional software you test once and trust it; with an LLM app you test many times and measure.
• Result: traditional tests pass or fail; LLM evals produce scores and quality ranges.

That last row is the mindset shift. An eval doesn't tell you the app is correct. It tells you how often it's good enough, on which kinds of input, and whether a change moved that number up or down.

Two categories: model evals and application evals#

Model evals test a model's general ability: reasoning, knowledge, maths and instruction following. AI labs such as OpenAI and Anthropic run them when they release a new model, for example a new Claude Sonnet or Opus, and publish the results with the release.

Application evals test the thing you build: a RAG system, an agent or a LangChain app. They answer a narrower and more useful question: does it work for your task and your users? This series focuses on application evals.

Example: a refund support bot#

Without evals: you ask your customer support bot five refund questions. It answers them well, so you ship it.

With evals: you build a dataset of 100 real customer refund and return queries, define the criteria (correct policy, polite tone, no invented promises), score every answer, and re-run the whole set after each prompt change. In this illustrative example, the eval shows the bot failing 18 "partial refund" questions, and you fix that before any customer sees it.

Here's the same idea as a tiny harness you can run. The bot is a fake function so the example works offline; in a real app it would call your prompt, model and retrieval. The scoring uses simple keyword checks, the most basic eval method there is.

import re

# 1. Dataset: realistic queries, each tagged with the policy a correct answer must state.
DATASET = [
    {"query": "Can I return shoes I bought two weeks ago?", "policy": "return_window"},
    {"query": "I opened the box but never used it. Full refund?", "policy": "return_window"},
    {"query": "I used half the bottle. Can I get some money back?", "policy": "partial_refund"},
    {"query": "One item in my order of three was missing.", "policy": "partial_refund"},
    {"query": "My order arrived broken. What now?", "policy": "damaged_item"},
    {"query": "The screen was cracked when it arrived.", "policy": "damaged_item"},
]

# 2. Success criteria, written down before looking at any answers.
POLICY_PHRASES = {
    "return_window": "within 30 days",
    "partial_refund": "partial refund",
    "damaged_item": "replacement",
}
POLITE = re.compile(r"\b(thanks|thank you|sorry|happy to help)\b", re.IGNORECASE)
PROMISES = re.compile(r"\b(guaranteed?|today|immediately)\b", re.IGNORECASE)


def support_bot(query: str) -> str:
    """A fake bot so the example runs offline. In a real app this calls your prompt and model."""
    q = query.lower()
    if "broken" in q or "cracked" in q:
        return "Sorry about that! We'll send a replacement at no cost."
    if "half" in q or "missing" in q:
        return "No problem, we'll give you a full refund today."
    return "Thanks for asking! You can return unused items within 30 days."


def grade(case: dict, answer: str) -> dict:
    """Score one answer against every criterion. True means the check passed."""
    return {
        "correct policy": POLICY_PHRASES[case["policy"]] in answer.lower(),
        "polite tone": bool(POLITE.search(answer)),
        "no invented promises": not PROMISES.search(answer),
    }


# 3. Run the system on the whole dataset and score every answer.
results = [(case, grade(case, support_bot(case["query"]))) for case in DATASET]

for criterion in results[0][1]:
    passed = sum(scores[criterion] for _, scores in results)
    print(f"{criterion:<22}{passed}/{len(results)}")

failing = [case for case, scores in results if not all(scores.values())]
print(f"\nFailing: {len(failing)} of {len(results)}")
for case in failing:
    print(f"  [{case['policy']}] {case['query']}")

Notice what the report gives you that five manual prompts never could: a score per criterion, and a list of exactly which cases failed. Change the bot, run it again, and you can see whether the numbers went up without anything else going down. A real model can answer differently on each run, so in practice you'd also run each case several times and track the pass rate.

The eval workflow engineers follow#

Evals aren't a one-off task before launch. They're a loop you keep running for the life of the app:

1. Define the target: pick the task or system to evaluate.
2. Set success criteria: decide what a good answer looks like.
3. Build a dataset: collect realistic test cases, including the awkward ones.
4. Choose an eval method: decide how each answer will be scored.
5. Run the system: execute it on the whole dataset, not a sample.
6. Analyse and improve: fix the model, prompt or system, then run again.
7. Deploy and monitor: add real production failures back into the dataset.

Steps 6 and 7 are loops. Every fix is checked by re-running the evals, and every production failure becomes a new test case, so your dataset gets stronger over time. LLM apps are probabilistic, so trust measurements, not impressions, and evaluate like it will fail in production.

Practice

Check your understanding

FAQ

Frequently asked questions