Lesson 7: Debugging bad answers
Every bad answer has exactly one first-failing stage. A fixed checklist finds it; guessing does not.
~15 minutes. Builds on Lesson 6. Reference: pipeline cheat sheet (checklist section).
Why a checklist
Barnett et al. catalogued the failure points of real RAG systems: missing content, missed top-k, wrong extraction, wrong format, incomplete answers - and found teams routinely fixing the wrong stage, usually by rewriting prompts when retrieval was the problem.[1] The cure is mechanical: walk the pipeline in order, stop at the first "no".
The checklist
- In the corpus? No: ingest gap. No downstream fix exists.
- In one self-sufficient chunk? No: re-chunk, or add the context line (Lesson 2).
- Filters match the row? No: fix extraction or metadata (Lesson 4) - remember, wrong filters produce plausible answers from the wrong set.
- Gold chunk in top k? No: hybrid, higher k, or query rewriting (Lesson 3).
- In top n after rerank, at an edge of the prompt? No: rerank/ordering (Lesson 5).
- Answer faithful, cited, complete? No: generation contract (Lesson 5), or context precision - too much noise in the prompt.
Each step is checkable because earlier lessons made it so: logged filter extractions (L4), retrieval traces with ranks (L3), citations per claim (L5), and gold-set numbers (L6). Debuggability was designed in, not bolted on.
Drill: symptom to stage
Classify each symptom into the stage family you would inspect first. Rerun until it is a clean sweep.
Retrieval practice
Your win: handed any bad answer plus its retrieval trace, you name the first failing stage in under a minute - and you never again "fix" a retrieval failure with prompt wording.
Primary source
Read Seven Failure Points When Engineering a RAG System - short, empirical, and the origin of half this checklist.
Stuck on a real bad answer? Paste the question, the retrieved chunks and the answer to your teaching agent and walk the checklist together.