Evaluation metrics

Retrieval metrics need only labels and arithmetic. Answer metrics need a judge. Know which is which.

Retrieval metrics (deterministic)

Defined over a gold set: questions with labelled relevant chunks. Canonical treatment: Introduction to Information Retrieval, ch. 8.

Recall@k
relevant chunks in top k / all relevant chunks. The ceiling metric: the generator cannot use what is not retrieved. Track at the k you actually pass to the generator.
Precision@k
relevant chunks in top k / k. Low precision means the prompt is mostly noise.
MRR
mean over questions of 1 / rank of first relevant chunk. 1.0 means always first; 0.5 means typically second.
nDCG@k
Rank-discounted gain versus the ideal ordering. Use when relevance is graded, not binary; otherwise recall@k + MRR suffice.

Answer metrics (LLM-graded)

From Ragas (metric docs):

Faithfulness
claims supported by retrieved context / all claims in answer. The hallucination detector.
Response relevancy
Does the answer address the question asked, fully and only.
Context precision / context recall
Was what got retrieved relevant / was everything needed retrieved, judged against a reference answer.
Judge caveats (Zheng et al.): position bias, verbosity bias, self-enhancement bias. Mitigate: swap positions and average, grade against a reference, never let a model grade its own family without checking agreement with human labels on a sample.

The loop

  1. Freeze a gold set (30+ real questions; grow it from production failures).
  2. Measure retrieval (recall@k, MRR) and answers (faithfulness, relevancy) on it.
  3. Change exactly one stage.
  4. Re-measure. Keep the change only if the numbers moved and nothing regressed.

One rule: never ship a retrieval or prompt change that was not measured on the gold set.