Module 12 — Evaluation: Measuring Whether It Works

This module is what truly separates professionals from dabblers. Beginners judge a RAG system by trying a few questions and going “looks good!” Experts measure. Without measurement you can’t tell if a change helped, and you’ll ship something that breaks quietly in production. If you internalize one mindset from this whole tutorial, make it this one.

1. Why “it looks good” is a trap

A demo with five hand-picked questions tells you almost nothing. Real users ask thousands of questions you didn’t anticipate, in messy wording, about edge cases. A system that nails your demo can fail half of real traffic. And when you tweak something — chunk size, a new embedding model, a reranker — “it feels better” is not evidence. You need numbers.

Rule: Build the way to measure quality before you try to improve quality. Otherwise you’re tuning blind.

2. Evaluate the two halves SEPARATELY

This is the key concept. A RAG system has two stages (retrieval, then generation), and a bad answer can come from either. You must measure them separately, or you won’t know what to fix.

Maybe retrieval failed — the right chunk was never fetched, so even a perfect model couldn’t answer.
Maybe retrieval succeeded but generation failed — the right chunk was right there and the model still answered wrong or made something up.

These need completely different fixes. Lumping them together leaves you guessing.

3. Retrieval metrics — did we fetch the right stuff?

These measure only the search part: given a question, did the relevant chunk(s) come back?

Recall@K (a.k.a. hit rate) — Was a relevant chunk somewhere in the top K we retrieved? This is the most important retrieval metric, because generation simply cannot use a chunk that was never fetched. If Recall@K is low, nothing downstream can save you.
Precision@K — Of the K chunks we fetched, how many were actually relevant? Low precision means lots of distractors crowding the context (which, per Module 10, hurts answers).
MRR / NDCG — rank-aware scores: not just whether a relevant chunk was retrieved, but how high up it was. (Higher = the best chunk is near the top, where rerankers and “lost in the middle” both want it.) You don’t need the formulas; know they reward putting the right chunk near position #1.

4. Generation metrics — given the context, was the answer good?

These assume retrieval did its job and measure the answer:

Faithfulness / groundedness — Is every claim in the answer actually supported by the retrieved context? This is your hallucination detector. An answer can sound great and still be unfaithful (it added facts not in the context).
Answer relevance — Does the answer actually address the question that was asked?
Context precision / recall — measures (popularized by tools like RAGAS) tying the answer back to which context was used and needed.
Correctness — compares the answer to a known-correct “gold” answer, when you have one.

5. The golden dataset — your most valuable artifact

To measure any of the above, you need a golden dataset (a.k.a. evaluation set): a curated list of

real questions + the correct answer + the source chunk(s) that contain the answer.

With this, you can automatically check: did retrieval fetch those source chunks (Recall@K)? Did the answer match the correct one (correctness)? Did it stay faithful?

Practical advice:

50–200 well-chosen examples beats vibes. You don’t need thousands to start.
Cover the range of real questions — common ones, edge cases, multi-part ones, and questions whose answer isn’t in the corpus (to check the system correctly says “I don’t know”).
Grow it from production failures. Every time the live system flubs a real question, add it to the set so you never regress on it again.

Building this dataset is the single highest-leverage thing you can own. It turns RAG from guesswork into engineering.

6. LLM-as-judge — measuring quality at scale

Checking faithfulness and relevance by hand doesn’t scale to hundreds of questions. The common solution: LLM-as-judge — use a capable model to grade answers (e.g. “Is this answer fully supported by this context? Score 1–5”).

Caveat: the judge isn’t perfect — it has its own biases. Calibrate it against human judgments on a sample so you trust its grades. Used carefully, it lets you evaluate continuously and cheaply.

Tools that package these metrics, tracing, and regression testing: RAGAS, TruLens, DeepEval, Arize Phoenix, LangSmith. You don’t need to memorize them — know that mature teams use a framework rather than ad-hoc spot checks.

7. How to actually improve — change one thing, measure, repeat

RAG tuning is empirical: intuition is often wrong, so test. The disciplined loop:

Measure your current system on the golden set (retrieval and generation metrics).
Change one thing (chunk size, embedding model, add a reranker, adjust K, tweak the prompt).
Re-measure. Keep the change only if the numbers improved.
Repeat.

Changing several things at once means you can’t tell what helped. One variable at a time.

8. The triage rule (memorize this)

When answers are bad, use your separated metrics to know where to look:

If Recall@K is LOW → fix RETRIEVAL. Work upstream: parsing quality, chunking (size/overlap/structure), add hybrid search, try a different embedding model, add metadata filters, raise K.

If Recall@K is HIGH but answers are still bad → fix GENERATION. Add or improve the reranker, strengthen the grounding prompt, fix context ordering (“lost in the middle”), cut distractor chunks, demand citations, add the “I don’t know” path.

This one rule turns “the answers are bad, help” into a precise, ordered debugging plan — and it’s exactly the kind of thinking that makes you valuable to solution architects.

Check yourself

Why must retrieval and generation be evaluated separately?
What is Recall@K and why is it the most important retrieval metric?
What does faithfulness detect?
What three things does each entry in a golden dataset contain?
Apply the triage rule: Recall@K is high but answers are wrong — what do you fix, and name two specific changes?

Next: Module 13 — Anti-Patterns →