Failure Points in LLM Applications: Where RAG Apps Break

Before you build evals: where can your app fail?#

This is part 2 of the LLM Evals series. In part 1, LLM Evals: Why "It Looks Good" Isn't Enough, you saw what evals are, why a few manual prompts aren't testing, and the workflow for running them. Before you build an eval for your own app, there's one question to answer first: how many places can it actually fail?

To answer it, we'll follow one customer and one question through one chatbot. Maya bought a jacket in a sale from an online clothing shop, and it doesn't fit. She asks the shop's help-centre chatbot: "Can I return a sale item?" The bot replies: "Yes, any unused item can be returned within 30 days for a refund."

Maya posts the jacket back. A week later the return is rejected, because the shop's policy says sale items are exchange only. That policy was in the help centre all along. So where did the bot go wrong? To find out, you need to look inside it.

How a RAG chatbot answers a question#

The help-centre bot is a RAG app: retrieval-augmented generation. Instead of answering from what the model memorised, it looks up the shop's own help articles first. Every request passes through four stages:

1. Query: the user's question, such as "How long do I have to return an item?"
2. Retriever: searches the help articles and returns the top k chunks, ranked by how relevant they look. This shop uses k = 5.
3. Generator: the LLM. It receives the query plus the retrieved chunks and writes a reply grounded in them.
4. Answer: what the user sees.

On a good day it works. For "How long do I have to return an item?", the retriever ranks the 30-day returns article first, the generator reads it, and the answer is "You have 30 days to return unused items." Right chunk, right answer.

Notice that there are two components doing very different jobs. The retriever's job is to find the right information. The generator's job is to use it. Either one can get its job wrong.

One eval pipeline or several?#

So here's the question to think about before you write any evals: for an app like this, do you need one eval pipeline or several? Consider the three ways the bot can give a bad answer:

• Retriever OK, generator broken: the right chunk is in the context, but the answer still gets it wrong.
• Generator OK, retriever broken: the right chunk never comes back, so even a perfect generator has nothing correct to work with.
• Both broken: the wrong chunks come back and the generator invents an answer anyway.

Now imagine a single pipeline that only checks the final answer. In all three cases it reports the same thing: bad answer. You know something failed, but not which part, and each case needs a different fix: better search or chunking for the retriever, a better prompt or model for the generator.

Example: the right chunk at position five#

Back to Maya. Here's her request going through the pipeline, with k = 5. In this example, the retriever returns these five chunks, best match first:

1. Returns: unused items within 30 days, full refund
2. Refunds go back to the original payment method
3. How to print a return label
4. Summer sale dates
5. Sale items can be exchanged but not refunded

The chunk with the real answer is there, in fifth place, the last one. The generator gets Maya's query plus all five chunks, leans on the general returns policy ranked first, and tells her she can return the jacket for a refund.

So who failed? The retriever did its job: you asked for the top five, and the right chunk was in the top five. The generator didn't: the answer was in front of it and it didn't use it. Retriever pass, generator fail. A final-answer check would only have said "wrong".

Evaluating the retriever and generator separately#

Here's a small sketch that checks each component on its own for four test cases. The retriever and the generator are stubs with fixed outputs, so it runs offline; in a real app they'd call your vector store and your LLM. Each test case records which help article holds the answer and one fact a good answer must state.

• Retriever eval: is the right article anywhere in the top k, and at what rank? This is a hit-at-k check, with the rank as a bonus.
• Generator eval: judged only on what the generator was given. If the right chunk was in its context, the answer must state the key fact. If it wasn't, the generator should say it doesn't know rather than guess.
• Diagnosis: the two results together name the component that failed.

K = 5  # the retriever returns the top five chunks

# Test cases: the help article that holds the answer, and the fact a good answer must state.
CASES = [
    {"query": "How long do I have to return an item?", "gold": "returns", "must_say": "30 days"},
    {"query": "Can I return a sale item?", "gold": "sale-items", "must_say": "exchange"},
    {"query": "Do you ship to Ireland?", "gold": "shipping", "must_say": "Ireland"},
    {"query": "Can I swap a gift card for cash?", "gold": "gift-cards", "must_say": "can't"},
]


# Stubs so the example runs offline. A real app calls a vector store and an LLM here.
def retriever(query: str) -> list[str]:
    """Return the ids of the top K help articles, best match first."""
    return {
        "How long do I have to return an item?": ["returns", "refunds", "labels", "exchanges", "sale-dates"],
        "Can I return a sale item?": ["returns", "refunds", "labels", "sale-dates", "sale-items"],
        "Do you ship to Ireland?": ["delivery-times", "returns", "labels", "refunds", "exchanges"],
        "Can I swap a gift card for cash?": ["refunds", "returns", "exchanges", "sale-dates", "labels"],
    }[query][:K]


def generator(query: str, chunks: list[str]) -> str:
    """Write the answer from the query and the retrieved chunks."""
    return {
        "How long do I have to return an item?": "You have 30 days to return unused items.",
        "Can I return a sale item?": "Yes, any unused item can be returned within 30 days for a refund.",
        "Do you ship to Ireland?": "I can't find that in our help centre. Please contact support.",
        "Can I swap a gift card for cash?": "Yes, gift cards are refunded to your original payment method.",
    }[query]


def eval_retriever(case: dict, chunks: list[str]) -> tuple[bool, int | None]:
    """Pass if the right chunk is anywhere in the top K, and report its rank."""
    if case["gold"] not in chunks:
        return False, None
    return True, chunks.index(case["gold"]) + 1


def eval_generator(case: dict, chunks: list[str], answer: str) -> bool:
    """Judge the generator only on what it was given."""
    if case["gold"] in chunks:  # it had the right chunk: did the answer use it?
        return case["must_say"].lower() in answer.lower()
    return "contact support" in answer.lower()  # it didn't: it should say so, not guess


def diagnose(retriever_ok: bool, generator_ok: bool) -> str:
    if retriever_ok and generator_ok:
        return "-"
    if retriever_ok:
        return "generator"
    return "retriever" if generator_ok else "both"


print(f"{'query':<40}{'final':<7}{'retriever':<12}{'generator':<11}failed")
for case in CASES:
    chunks = retriever(case["query"])
    answer = generator(case["query"], chunks)
    final_ok = case["must_say"].lower() in answer.lower()  # the one-pipeline view
    retriever_ok, rank = eval_retriever(case, chunks)
    generator_ok = eval_generator(case, chunks, answer)
    r = f"pass #{rank}" if retriever_ok else "fail"
    print(
        f"{case['query']:<40}{'pass' if final_ok else 'fail':<7}{r:<12}"
        f"{'pass' if generator_ok else 'fail':<11}{diagnose(retriever_ok, generator_ok)}"
    )

Read the table row by row. Maya's question fails the final-answer check, but the retriever found her chunk at rank 5, so the generator is to blame. The shipping question is the opposite: the retriever missed, and the generator correctly said it couldn't find the answer. The gift card question fails both ways. A real eval would use many more cases and a softer answer check, such as an LLM judge, but the structure stays the same.

Three levels of failure#

Maya's bot failed at one component. Step back, and an LLM application can fail at three different levels, and your evals need to cover each one:

• Component level: a single part breaks. The prompt, the retriever, or the re-ranker that reorders retrieved chunks before the generator sees them. Evaluate each component with its own test cases, as in the sketch above.
• Workflow level: every part works on its own, but the flow of the pipeline breaks. A step runs in the wrong order, or one stage hands the next something it can't use. Evaluate the whole pipeline end to end.
• Application level: the system as a whole has to work well for real users. Is the latency good? Does the whole system hold up under real traffic, not just one answer at a time?

So there isn't just one place where an LLM app can go wrong. There are failure points at every level, and good evals are designed with all of them in mind, so when something breaks, you know exactly where to look.

Practice

Check your understanding

FAQ

Frequently asked questions