An LLM evaluation dataset is a versioned set of tasks, context and judgment criteria used to decide whether an AI application is ready to ship. It is not merely a spreadsheet of prompts and ideal prose. A useful dataset represents real product behavior, known failures, important user groups and the costs of getting an answer wrong.

This guide builds the data layer beneath AI Testing & Evaluation and LLM regression testing.

Golden dataset, development set and production sample

Separate three related assets:

• Golden dataset: reviewed cases used for release decisions and stable comparisons.
• Development set: cases writers and engineers can use while iterating.
• Production sample: privacy-reviewed examples used to detect new intents and failures.

If people repeatedly tune prompts against the golden set, it stops behaving like an independent test. Keep part of the release set access-controlled or rotate reviewed cases when necessary.

Define the decision before collecting examples

Write down what the evaluation will decide:

• Is a new model good enough to replace the baseline?
• Did a prompt change damage a high-risk workflow?
• Does a RAG system answer from approved evidence?
• Can an agent select and execute tools within its permissions?
• Is the system ready for a limited canary or production release?

Then define failure severity. A stylistic preference and an unauthorized refund action must not contribute equally to one average score.

Design a durable case schema

Each row should contain enough information to reproduce and interpret the result:

{
  "id": "billing-refund-017",
  "input": "Refund my subscription from last month",
  "context_refs": ["refund-policy-v8"],
  "expected_facts": ["refund window is 14 days"],
  "allowed_tools": ["lookup_subscription"],
  "forbidden_actions": ["issue_refund"],
  "criteria": ["cites policy", "requests approval for exception"],
  "risk": "high",
  "locale": "en-BE",
  "source": "redacted_production_failure",
  "label_version": 3
}

Use references and criteria only when they are truly known. For open-ended ideation, a single “reference answer” can incorrectly punish valid alternatives.

Collect production examples carefully

Production interactions reveal real wording, context lengths and failure modes. Sample across meaningful slices rather than taking the easiest recent requests:

• common and rare intents;
• successful and failed workflows;
• short and long context;
• languages and locales the product supports;
• new users and experienced users;
• read-only and consequential agent actions;
• retrieval, tool-use and no-tool cases.

Feedback signals are useful but imperfect. A thumbs-down may reflect latency or UI friction rather than answer quality. Review the complete trace before assigning a label.

Use human labels where judgment matters

Human labeling requires a rubric and quality control. Define:

• who is qualified to judge the task;
• what evidence raters may use;
• examples of passing, borderline and failing output;
• how disagreements are resolved;
• which errors are automatic release blockers.

Give overlapping cases to multiple raters when the decision is subjective or high-impact. Track disagreement rather than averaging it away. Disagreement can reveal an unclear product requirement or a rubric that asks raters to infer missing context.

Add synthetic cases without replacing reality

Synthetic examples help expand sparse failure categories:

• prompt-injection variations;
• missing or contradictory context;
• malformed tool results;
• unusual but valid input formats;
• multilingual paraphrases;
• boundary values and long inputs;
• tool timeouts and partial failures.

Treat model-generated cases as proposals. Review them for duplicates, impossible scenarios, leaked answer patterns and assumptions that do not exist in the product. Preserve their source so synthetic and production-derived results can be reported separately.

Build edge and adversarial slices

Do not hide hard cases inside a global average. Create named slices for:

• unsupported questions where the correct behavior is uncertainty;
• conflicting retrieved documents;
• instructions embedded in untrusted content;
• invalid structured outputs;
• repeated agent actions and loops;
• permission escalation attempts;
• provider failures and interrupted streams;
• previously confirmed production incidents.

The Structured Outputs foundation and JSON parsing guide provide deterministic cases for response contracts.

Match criteria to the task

Prefer objective graders wherever possible:

Requirement	Suitable evidence
Valid response contract	JSON Schema or typed validator
Correct tool use	Tool name, arguments and resulting state
Grounded answer	Required source and supported claims
Code change	Tests, linting and reviewed diff
User-facing quality	Task-specific rubric and calibrated judge
High-impact decision	Qualified human review

Do not turn every requirement into a 1–5 score. Booleans, labels and hard constraints are often clearer. Use LLM-as-a-Judge only for criteria that genuinely need model judgment.

Protect privacy and security

An eval dataset can become a concentrated copy of user conversations, documents and tool traces. Before collecting production data:

1. establish a lawful purpose and retention rule;
2. minimize fields to what the evaluation needs;
3. remove direct identifiers and secrets;
4. restrict access and audit exports;
5. separate raw samples from reviewed, sanitized cases;
6. confirm what an external evaluation provider stores and where;
7. delete cases when the source data must be removed.

Redaction can change task meaning, so validate sanitized cases again. Synthetic replacements may be safer when exact production text is unnecessary.

Version every dependency

Version the dataset separately from:

• prompts and tool descriptions;
• model configuration;
• retrieval corpus and index;
• graders and rubrics;
• application code.

Record a changelog explaining which cases were added, relabeled or retired. Never silently edit historical results after a label changes; rerun the affected evaluation with the new dataset version.

Connect the dataset to release gates

Run a small affected slice on ordinary pull requests and the broader golden set before a protected release. Save per-case results, not only the mean. The GitHub Actions foundation explains how to preserve artifacts and protect credentials, while AI Operations connects offline evidence to canaries and rollback.

A dataset is ready when it supports an actual decision, represents the workflows and risks that matter, and can be reproduced without exposing more user data than the evaluation requires.