How to Build an Evaluation Set — Proving Your RAG, Text-to-SQL, and Agent Improvements with Numbers

Summary — The previous two posts (Prompting, RAG, or Fine-Tuning? and How to Serve Internal Database Data to an LLM) repeatedly said “first build 30–50 evaluation questions.” This post covers that evaluation set: where to source questions, how to define answers and grading criteria, how many you need, how to grade, how to measure each pipeline stage separately, and how to keep it running. An evaluation set is not about the model; it is what gives your team the right to say “it got better.” (The scenario is generalized and fictional, and example figures are for illustration.)

0. The conclusion first: a one-page summary

The core principle: without an evaluation set, improvement is a feeling; with one, it is evidence.

1. The story continues — the trap of “it seems better”

In a few weeks, the team from the previous posts changed a lot. They refined the prompt, connected policy documents with RAG, moved DB questions into per-domain flows, and added a classification model to the orchestrator. A weekly meeting goes like this:

• Developer: “After the prompt change, the answers feel much more natural.”
• Business owner: “But yesterday I asked about the leave carry-over policy and got the wrong number.”
• Team lead: “So did it get better or not?”

Nobody can answer. They changed many things but never measured before and after against the same standard. When one good case looks like improvement and one bad case looks like regression, you cannot tell which change helped and which broke something else.

2. What an evaluation set is and why you need it first

An eval is a test in which you give an AI system an input and apply grading criteria to its output to measure success. A collection of items — input plus answer criteria — is an evaluation set. Published guidance on agent evals (Anthropic) defines evals the same way and distinguishes capability evals, for improving features, from regression evals, for preventing backsliding.

When a capability item’s pass rate becomes high enough, promote it to the regression set. If a set keeps producing the same score it no longer signals improvement (saturation), so add harder items.

Keep the evaluation set separate from training and example data. If a question used as a few-shot example in the prompt or as fine-tuning data is also used for evaluation, the system has “seen the test,” and the score is inflated. Set evaluation items aside.

3. Where do the questions come from?

You do not need a perfect set from the start. Anthropic’s guidance recommends starting with 20–50 simple tasks and drawing them not from imagination but from the manual checks and real failures you already have.

Balance matters too. Measure not only whether the system “acts when it should” but also whether it “does not act when it should not.” For a DB assistant, include items on whether it refuses out-of-scope questions and blocks requests for data the user may not see. Measuring one side optimizes one side (like a system that searches for everything).

4. What makes a good evaluation item

The most important criterion is “could two domain experts reach the same verdict?” An item on which experts split between pass and fail for the same answer is ambiguous. If a system keeps failing an item at 0%, suspect that the item is wrong before blaming the system.

One item, structured (example)

{
  "id": "hr-014",
  "area": "HR",                      // domain (for classification eval)
  "type": "db_number",               // doc_lookup | db_number | mixed | out_of_scope | safety
  "risk": "high",                    // cost of error (decision-making use)
  "question": "What is team A's leave usage rate this month?",
  "expected": {
    "sql_result": [{"dept": "A", "rate": 0.42}],   // result of the verified SQL
    "must_cite": ["leave_policy_v3"],              // if a document source is required
    "should_refuse": false
  },
  "notes": "Usage rate follows the semantic layer's 'leave usage rate' definition"
}

Mix easy, medium, and hard items. Without hard items, the set saturates quickly.

5. How many do you need? — error in numbers

“20–50 is enough to start” is a standard for starting. If a pass rate comes out at 80%, how far you can trust it depends on the number of items. The approximate width of a 95% confidence interval by normal approximation is as follows.

How to read it – On 50 items, “78% → 82%” can be fluctuation within the error. Do not declare a small difference an improvement. – A paired comparison — before and after on the same items — has less error than comparing different items. That is why you always use the same set before and after a change. – There is a proposal to report evaluation results with their uncertainty (Miller, 2024). If two compared scores overlap within their error ranges, it is hard to say there is a difference. – Adding items is not the only answer. Splitting by domain shrinks the items per domain, widening the error again. Build more items for high-risk domains.

(The widths above are a normal approximation for a simple proportion; with few items or extreme pass rates, other methods are more accurate.)

6. Grading — code, model, or human

What to know when using model grading – Research (Zheng et al., 2023) reports that strong model judges agree with human preferences over 80%, similar to the agreement between humans. It also analyzes limits such as position bias, verbosity bias, and self-preference bias. Mitigate by grading twice with the order swapped, or by stating criteria explicitly so length does not sway the judge. – Giving the judge an “Unknown” option reduces forced verdicts. – Calibrate model grading by periodically comparing it with human-graded samples. Where it disagrees with humans on a type of item, fix the wording of the grading criteria. – Grade the outcome, not the path. There may be more than one way to reach a correct answer. Give partial credit for multi-part tasks.

7. Measure stage by stage

A single end-to-end accuracy does not tell you where the problem is. Following the structure of the previous posts, separate the stages and give each its own metric.

• nDCG@10 is the main metric of the retrieval benchmark BEIR, chosen because it is rank-aware and handles both binary and graded relevance. But public benchmarks like BEIR are sanity checks; for real decisions, a small evaluation set from your own domain matters more.
• The RAG evaluation framework Ragas proposes a suite of metrics that measure, without ground-truth labels, whether retrieval found focused relevant passages, whether generation is faithful to them, and how good the generation is. Such automatic metrics are useful for quick checks, but it is safer to confirm final judgments against human-written answer items.
• If you use a decision model in the orchestrator, keep a separate classification evaluation set and verify on your own data that its probabilities match actual accuracy (calibration).

8. When results differ every time — handling non-determinism

An LLM can give a different answer to the same question each time, so one run is not enough. Two metrics try the same item multiple times (k times).

For example, even if a single try succeeds 90% of the time, assuming tries are independent, the chance that all five succeed is 0.9 to the fifth power, about 59%. For a service where users expect stable answers every time, pass^k is closer to what they actually feel.

Also build a stable evaluation harness. Each trial should start from a clean state so traces from earlier trials do not distort results.

9. Operations — not a one-time job

1. Run the same set at every change. Whether you changed the prompt, model, chunking, thresholds, or flows, compare before and after on the same set. Change one thing at a time to know the cause.
2. Classify failures by cause. Classification error / retrieval miss / ignored evidence / column misreading or join error / refusal or permission misfire / grading error. The classification is your next improvement backlog.
3. Read the transcripts yourself. Check whether a failure is a real system mistake or a grader rejecting a valid answer. If a failure looks unfair, fix the evaluation before trusting the score.
4. Add harder items as pass rates rise. If you stay at 100%, the signal disappears.
5. Name an owner. An evaluation set goes stale as the system changes unless someone maintains it. Open a path for business and product people to contribute items.
6. Turn new failures into items. Add wrong answers found in operation to the set right away so the same mistake does not return.
7. Handle confidentiality. Generalize and anonymize personal data and company secrets in items taken from real questions before storing them.

10. A worked example — a 60-item set

A first evaluation set for the internal AI assistant of the previous posts could be composed like this. (The allocation of items is illustrative.)

A hypothetical way to read results (illustrative figures)

Viewed stage by stage like this, you see not the impression that “it got better after the prompt change” but which change moved which metric by how much. An unchanged item (faithfulness) tells you the next improvement task.

11. Checklist before you start

• [ ] The evaluation set is separate from training and example data
• [ ] Items came from real questions, failure records, and interviews, and are anonymized
• [ ] Answer criteria are clear enough for two experts to reach the same verdict
• [ ] Items carry tags for area, type, and risk
• [ ] Items include both “when it should act” and “when it should not”
• [ ] DB questions are graded on execution results, not SQL strings
• [ ] Code grading is the default, and model grading is calibrated against human samples
• [ ] Stage metrics (classification, retrieval, generation, DB, safety) are viewed separately
• [ ] Results are reported with an error range, and small differences are not declared improvements
• [ ] You compare before and after on the same set at every change, and add harder items as pass rates rise
• [ ] The set has an owner

12. Wrap-up

• An evaluation set is the right to say “it got better,” not a model feature. Build it before improving.
• Start with 20–50, but detecting small differences needs more, and you must look at the error range too.
• Define answers so two experts reach the same verdict, and grade DB questions by execution result.
• Code grading is the default; model grading knows its biases and is calibrated with human samples; humans handle samples and disputes.
• Measure classification, retrieval, generation, DB, and safety separately rather than one end-to-end score, and you can see where to fix.
• Evaluation is operations, not a one-off. Run it at every change, classify failures, and turn new failures into items.

Coming next: implementing the orchestrator and domain flows — classification prompts, confidence thresholds, and out-of-scope handling.

13. References

The evidence in this post is the author’s summary and interpretation of the sources below. Please check the original sources for exact figures and wording.

• Anthropic, Demystifying evals for AI agents — definition of evals, capability vs. regression evals, starting with 20–50, grading methods, pass@k and pass^k, roadmap
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 — agreement of model judges with human preferences, biases
• Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation, EACL 2024 — a metric suite for RAG evaluation without ground-truth labels
• Thakur et al., BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021 — retrieval evaluation such as nDCG@10
• Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024 — error ranges for evaluation results
• Li et al., BIRD: Can LLM Already Serve as A Database Interface?, NeurIPS 2023 — execution-result-based Text-to-SQL evaluation

The methods here are based on generally published practical guidance and the author’s synthesis, and example figures are for illustration. Validate against your own data and risk level before adopting.