Lesson 5: Rerank, order, generate

Retrieve wide and cheap, then rerank narrow and precise, then hand the generator a prompt it cannot misread.

~15 minutes. Builds on Lesson 4.

Two stages, two budgets

First-stage retrieval (Lessons 3-4) scores query and chunk separately - that is what makes it fast enough to scan millions of chunks. A reranker reads query and chunk together (a cross-encoder), which is far more accurate and far too slow for the whole corpus. So: retrieve ~100 candidates cheaply, rerank them, keep the best ~10.^[1] ColBERT sits between the two designs - late interaction, near cross-encoder quality at a fraction of the cost.^[2]

candidates = hybrid_search(query, filters, top=100)   # recall-oriented
kept       = rerank(query, candidates)[:10]            # precision-oriented

The stages compound: contextual retrieval alone cut failed retrievals 49%; adding reranking took it to 67%.^[3]

Order is not cosmetic

Liu et al. showed models use information at the start and end of the context well and degrade sharply on information buried in the middle - even models with plenty of window.^[4] Consequences:

• Place the strongest chunks first and last; weaker support goes in the middle.
• More chunks is not safer: every marginal chunk pushes something toward the dead middle and dilutes context precision. Keep n small and measured.

The generation contract

prompt =
  system: "Answer only from the numbered sources. Cite as [n] after each claim.
           If the sources do not contain the answer, say so - never guess."
  sources: [1] chunk, [2] chunk, ...   # reranked, strongest first and last
  question

Three clauses do the work: only from sources (grounding), cite per claim (auditability - and citations make the eval in Lesson 6 checkable), "not found" over guessing (the honest failure mode). A fluent invented answer is strictly worse than a refusal: the refusal is visible, the invention is not.

Retrieval practice

From memory, before scrolling up.

Primary source

Read Lost in the Middle (at least sections 1-3): it is the paper that turns prompt assembly from taste into engineering.

Curious whether a reranker earns its latency on your data? Ask your teaching agent to help design the A/B on your gold set after Lesson 6.