LLM regression testing compares a proposed AI-system configuration with a known baseline on the same representative tasks. It detects when a prompt, model, retrieval pipeline, tool or policy change makes the product worse—even when every request still returns HTTP 200.

This is not an exact-string test. It is a controlled experiment with versioned inputs, explicit graders and release thresholds. Use the broader AI testing guide to decide which behavior belongs in deterministic tests and which requires evaluation.

What can cause quality drift?

Treat the complete AI system as the test subject:

Change	Possible regression
Prompt or examples	Missing instructions, style changes, worse edge cases
Model or snapshot	Different reasoning, refusals, tool use, latency or cost
RAG documents	Stale, contradictory or missing evidence
Chunking and ranking	Relevant context no longer reaches the model
Embedding model	Changed nearest neighbours and recall
Tool schema	Invalid arguments or wrong tool selection
Orchestration	Duplicate actions, lost state or broken fallbacks
Safety policy	Over-refusal or newly permitted unsafe behavior

A useful regression report identifies which component changed. “Average quality fell” is less actionable than “retrieval recall fell for policy questions after re-indexing.”

Build a versioned baseline

Store a manifest for each approved baseline:

baseline: support-agent-2026-08-20
dataset: support-eval-v12
prompt: support-system-v9
model: pinned-provider-model-snapshot
retrieval_index: help-center-v34
embedding_config: embeddings-v5
tool_schema: support-tools-v7
grader_config: support-rubric-v4

Pin what the provider allows you to pin and record everything else. If an upstream model changes behind a stable alias, the manifest still shows that your own prompt and retrieval configuration did not change.

Design the evaluation dataset

Use stable case identifiers and meaningful slices:

• routine versus difficult tasks;
• supported versus unsupported questions;
• language, locale or customer tier;
• short versus long context;
• retrieval-required versus model-only tasks;
• read-only versus consequential agent actions;
• known incidents and previous regressions.

Keep a locked comparison set for release decisions and a separate development set for iteration. Repeatedly tuning against the same release set can overfit prompts and graders to the benchmark.

For RAG systems, version both the question and its expected evidence. A response can sound correct while citing the wrong document. Measure retrieval and generation separately:

1. Did the retriever return the required source?
2. Did the model use that source faithfully?
3. Did the answer cite or expose evidence as required?

Use a grader stack, not one score

Combine signals according to the task:

type EvalResult = {
  schemaValid: boolean;
  requiredFactsPresent: boolean;
  citationsSupported: boolean;
  toolCallCorrect: boolean;
  rubricScore?: number;
  latencyMs: number;
  inputTokens: number;
  outputTokens: number;
};

Prefer deterministic graders for schema, tool arguments, executable code, citations and state changes. Use model graders for qualities that genuinely require judgment, then calibrate them against human-reviewed examples.

Track hard failures separately. An invalid permission boundary should fail the release even if the average helpfulness score improves.

Compare candidates fairly

Run baseline and candidate configurations against the same cases under comparable conditions. Because model output varies, repeat sensitive cases or use enough representative cases to understand variance.

Report:

• overall pass rate and score distribution;
• results for each risk or intent slice;
• new failures and fixed failures;
• cost and latency changes;
• grader disagreement;
• traces for consequential agent scenarios.

Avoid a universal “five percent” threshold. Set gates from business impact and measurement noise. A small decline in casual copywriting may be acceptable; one new unauthorized tool action is not.

Test prompt and model changes

For a prompt change, show the exact diff and run the affected slices. For a model migration, broaden the suite to include:

• instruction following;
• structured output and tool selection;
• long-context behavior;
• safety and refusal boundaries;
• latency, throughput and token use;
• fallback compatibility.

Do not promote a model solely because a public benchmark improved. The decision dataset should represent the application’s actual tasks.

Test RAG and embedding changes

Changing embeddings, chunk sizes or ranking can silently change evidence selection. Preserve a retrieval test set with known relevant document IDs and measure whether the required source appears among the first results.

Then evaluate the generated answer using the retrieved evidence. This split tells you whether a failure belongs to retrieval or generation.

When documents change, record the corpus version and decide whether expected answers must change too. Updating the expected answer is a reviewed product decision, not a way to make a failing test green.

Test agent workflows

Regression tests for agents need trace-level assertions:

• permitted tool selected;
• arguments validate against the schema;
• approval requested at the correct boundary;
• retry did not duplicate the action;
• failure led to a safe recovery path;
• final state matches the intended outcome.

Run destructive tools against fakes or disposable environments. Link permission and credential controls to the AI Security foundation and repository-change controls to Coding Agents.

Add evaluations to CI/CD

Use a tiered pipeline:

1. Every pull request: deterministic tests and a small affected eval set.
2. Protected merge or scheduled run: broader dataset, repeated samples and model graders.
3. Before release: risk slices, latency/cost budgets and human review where required.
4. After release: canary traffic, monitored quality signals and rollback rules.

Do not expose provider or production credentials to untrusted pull requests. Store evaluation artifacts—the manifest, summary, per-case result and relevant traces—so reviewers can inspect why a gate passed.

The GitHub Actions foundation covers protected workflows and deployment gates. AI Operations connects offline results with canaries, monitoring and incident response.

Keep the suite useful

Evaluation suites decay unless they are maintained:

• add confirmed production incidents as regression cases;
• retire or update cases only through review;
• monitor duplicated and low-information examples;
• recalibrate model graders when models or rubrics change;
• keep sensitive data minimized and access-controlled;
• investigate flaky cases instead of hiding them with retries.

The output of regression testing is not merely a red or green badge. It is evidence showing what changed, which users or workflows are affected, and whether the team should ship, revise or roll back the candidate.