A RAG answer can be wrong for two very different reasons: the system may fail to retrieve useful evidence, or the language model may fail to use evidence it received. If you judge only the final answer, it is difficult to tell which part to fix.
Start by evaluating retrieval on its own. You do not need a large benchmark to find obvious failures. A carefully assembled set of representative questions can reveal whether the right source material is reaching the model.
Collect questions people might actually ask about the documents you plan to search. Include several shapes:
For each answerable query, record one or more source passages that contain the evidence. Keep the expected evidence narrow enough that a person can judge whether a retrieved result is relevant. For an unanswerable query, record that no supporting passage exists; otherwise a system can look good by retrieving something loosely related.
This set is a diagnostic fixture, not an objective measure of all future use. Make the source documents and test questions representative of the intended users, and note where the set is thin.
Inspect the top k retrieved passages for each query. Two useful questions are:
For a small test set, a manual table can be enough:
| Query | Expected evidence | Relevant in top 3? | Rank of first relevant result | Notes |
|---|---|---|---|---|
| Direct lookup | Password policy, section 2 | Yes | 1 | Exact phrase match |
| Paraphrase | Password policy, section 2 | Yes | 2 | Semantic match |
| Multi-part | Sections 2 and 5 | No | — | Only one part retrieved |
| Unanswerable | None | — | — | Similar but unrelated result |
Do not start by declaring that one metric tells the whole story. Recall at a chosen k is easy to interpret, but it treats every relevant passage as equally useful. Rank-sensitive measures can help when result order matters. The manual notes often explain a failure more clearly than a single average.
Once you can reproduce a retrieval result, change one setting at a time. For example:
Then test other factors separately: overlap, metadata filters, keyword search, embedding model, number of retrieved chunks, or a reranker. If you change chunking, the embedding model, and the prompt at once, a better answer will not tell you which change helped.
Chunking is a hypothesis about what context should travel together. A table row, a procedure step, and a long prose section may need different treatment. Try to preserve meaningful boundaries; splitting every document at a fixed character count can separate a heading from the explanation it introduces.
After retrieval is measured, evaluate answer generation separately. For each test query, retain the exact retrieved passages and inspect whether the answer:
This gives you a useful two-stage diagnosis. If the expected passage never arrives, work on retrieval or source coverage. If it arrives but the answer is still unsupported, work on the prompt, output format, or model behavior.
Do not build a test set from only simple questions. Add examples with stale documents, duplicate pages, near-matching terms, metadata boundaries, and missing information. If the system must respect access controls, include queries for which the user should not receive a particular document; test authorization separately from semantic relevance.
Save the query set, retrieved document IDs, ranking, chunk configuration, and date of the run. That makes a later comparison meaningful when you update an embedding model or alter the corpus.
A small hand-labeled set is a practical debugging instrument. It can tell you that a specific change fixed a known failure. It cannot prove that retrieval works for every user, every phrasing, or every future document. As new failures appear, turn them into additional test cases and periodically check whether the set still represents real usage.
The best first RAG evaluation is not a perfect benchmark. It is a repeatable way to see what evidence the model actually receives.
You might also be interested in reading these related topics
