Reranking & RAG Evaluation
Retrieval gets you candidates fast. Reranking gets the right ones to the top. Evaluation tells you if any of it worked.
Advanced
TopNotchNote
Notes that matter
why rerank after retrieval
Vector similarity search (a "bi-encoder": query and document embedded independently, then compared) is fast enough to run over millions of chunks, but the independence that makes it fast also makes it a coarser relevance signal. A cross-encoder reranker takes the query and a candidate chunk together as joint input and scores relevance directly — much more accurate, but too slow to run over a whole corpus. The standard pattern: retrieve a wide candidate set cheaply (e.g. top 50 by vector similarity), then rerank just those with the expensive cross-encoder down to the top 5-10 that actually go in the prompt.
candidates = vector_search(query, k=50) # cheap, broad recall
reranked = cross_encoder.rank(query, candidates) # expensive, narrow precision
top_chunks = reranked[:8] # what actually enters the promptevaluating retrieval in isolation
| Metric | Answers |
|---|---|
| Recall@k | Of the chunks actually relevant to the query, what fraction appear in the top k retrieved? |
| Precision@k | Of the top k retrieved chunks, what fraction are actually relevant? |
| MRR (Mean Reciprocal Rank) | On average, how high does the first relevant chunk rank? |
These require a labeled evaluation set of (query, relevant chunk IDs) pairs — without ground truth, you can't distinguish "the model answered wrong" from "retrieval never surfaced the right chunk in the first place."
evaluating the full RAG output
| Metric | Checks |
|---|---|
| Faithfulness | Is every claim in the generated answer actually supported by the retrieved context, or did the model add unsupported claims? |
| Answer relevancy | Does the answer actually address the question asked, independent of whether it's factually grounded? |
| Context precision/recall | Retrieval quality, scored automatically instead of by hand |
Frameworks like RAGAS compute these by using a second LLM as a judge — e.g. asking a model whether each sentence in the answer is entailed by the retrieved context. That makes evaluation scalable, but the judge model itself needs periodic spot-checking against human judgment; see Evaluating LLM Applications for the LLM-as-judge pattern in general.
where to go from here
Evaluating LLM Applications — the LLM-as-judge pattern used by most RAG eval frameworks.
Chunking & Retrieval Strategies — the upstream decisions reranking is compensating for.
RAG Architecture — the full pipeline this page's two stages plug into.
Chunking & Retrieval Strategies — the upstream decisions reranking is compensating for.
RAG Architecture — the full pipeline this page's two stages plug into.