Lesson 6: Measuring quality

A gold set, four numbers, one loop. Every stage decision so far becomes testable here.

~18 minutes. Builds on Lesson 5. Reference: metric formulas.

The gold set

A gold set is 30+ real questions, each labelled with (a) the chunks that should be retrieved and (b) the facts a correct answer must state. Source them from real agent questions and, forever after, from production failures - every bad answer that reaches you becomes a gold row. Without it, every tuning decision in Lessons 2-5 is superstition.

Retrieval numbers first - they are free

Recall@k and MRR need no model, no judge - just the labels and arithmetic (definitions: IR-book ch. 8^[1], formulas in the reference). Recall@k is the ceiling: an answer cannot cite a chunk retrieval never returned. Measure it at the k you actually pass to the generator.

Answer numbers need a judge - calibrate it

Ragas-style metrics - faithfulness, response relevancy, context precision/recall - use an LLM to decompose answers into claims and check them against the context.^[2] Useful, but the judge itself is biased: position bias, verbosity bias, self-enhancement bias are all documented.^[3] Working rules: grade against the gold reference (not pairwise where position bias lives), spot-check a sample of judge verdicts by hand before trusting a metric, and never conclude from judge scores alone that a change "improved quality".

The loop

for each proposed change (new chunking, hybrid weights, reranker, prompt):
  baseline = measure(gold_set)          # recall@k, MRR, faithfulness, relevancy
  apply the ONE change
  after = measure(gold_set)
  keep it only if targets moved and nothing regressed

Retrieval practice

From memory, before scrolling up.

Primary source

Read IR-book chapter 8 for the deterministic metrics, then the Ragas metric docs with Zheng et al. open beside them as the antidote.

Ask your teaching agent to help draft your first 30 gold questions from a real query log - it is an hour of work together.