A canary deployment sends a controlled share of eligible traffic to a candidate AI configuration while the approved baseline remains available. It limits blast radius and tests assumptions against production conditions. It does not replace offline evaluation, security review or a rollback plan.

Use canaries for changes to models, prompts, retrieval, tool use and orchestration when real traffic can reveal behavior the test set does not cover. The AI Testing & Evaluation hub defines the evidence required before traffic reaches the candidate.

What is the deployment unit?

Version the complete behavior being released:

{
  "release": "support-agent-2026-08-24.1",
  "model": "provider-model-snapshot",
  "prompt": "support-v14",
  "retrieval_index": "help-center-v38",
  "embedding_config": "embeddings-v6",
  "tool_schema": "support-tools-v9",
  "policy": "agent-policy-v5",
  "application_commit": "abc123"
}

Without this manifest, a quality change cannot be attributed reliably. “The canary model” may actually differ in retrieval corpus, prompt and tool policy too.

Gate the canary offline first

Before production exposure:

1. run deterministic contract, permission and security tests;
2. compare candidate and baseline on the same evaluation dataset;
3. inspect high-risk slices and severe failures;
4. verify latency and cost budgets;
5. test observability, assignment and rollback;
6. obtain the required human approval.

Use LLM regression testing for prompt, model, RAG and embedding comparisons. A candidate with a known critical regression should not use production users as a test harness.

Assign traffic consistently

Use a stable, privacy-appropriate assignment key so the same eligible user, tenant or conversation remains in one variant. Random assignment per request can mix state, retrieval caches and agent behavior inside one workflow.

Decide eligibility before the percentage:

• exclude high-risk workflows initially;
• keep regulated or contractually restricted data on approved paths;
• avoid splitting one stateful agent run across configurations;
• identify internal and test traffic separately;
• preserve an emergency override to stable.

The starting share and expansion schedule should reflect risk and traffic volume, not a universal percentage template.

Instrument candidate and baseline equally

Every request should record the release identity and enough privacy-safe context to compare:

• eligible population and assignment;
• task or intent slice;
• model, prompt, retrieval and tool versions;
• provider errors and retries;
• latency and token usage;
• tool calls, approvals and final state;
• user feedback and verified outcome where available;
• fallback or rollback events.

Do not log raw prompts, retrieved documents or secrets by default. Keep the identifiers required for diagnosis while minimizing sensitive content.

Define promotion and rollback signals

Use a mix of hard and trend-based gates.

Immediate stop examples:

• authorization or tenant-isolation failure;
• unsafe or duplicate side effect;
• corrupted state or migration incompatibility;
• severe policy violation;
• missing telemetry that makes the canary unobservable.

Comparative signals:

• verified task success;
• model and application error rate;
• fallback and retry use;
• latency and cost per successful task;
• human takeover or correction;
• quality scores on reviewed production samples;
• retrieval and tool failures by slice.

Set thresholds from baseline variance, business impact and sample size. Avoid promoting from a handful of requests or one noisy average.

Separate canary rollout from A/B experimentation

A canary asks: Can the already-approved candidate be expanded safely? An A/B experiment asks: Which experience performs better?

Do not keep a risky candidate live merely to obtain statistical significance. Canary rollback protects users; experimentation optimizes a product after minimum safety and quality requirements are met.

Model and prompt changes

Evaluate instruction following, refusals, structured output, tool selection, latency and cost. Sample real outputs for high-value intents and compare them using calibrated graders plus human review.

Keep the baseline immediately routable. If a provider alias changes outside your deployment, create a new observed release identity and rerun the appropriate evaluation rather than silently treating it as the old baseline.

RAG and embedding changes

Monitor retrieval and generation separately:

• whether required evidence appears in retrieved results;
• source freshness and permission filtering;
• answer support and citation behavior;
• latency and cost of retrieval;
• empty, contradictory or stale context;
• changes by document type and intent.

A generated answer may remain fluent while retrieval recall deteriorates. The AI Application Architecture foundation helps isolate these boundaries.

Agent and tool changes

For agents, a canary must preserve one workflow version from start to finish. Track:

• tool selection and argument validity;
• approval and permission enforcement;
• repeated calls and loop termination;
• recovery from tool failures;
• consequential side effects;
• cost and duration per successful task.

Start with read-only or reversible tasks. Do not use a canary percentage as the only control over destructive permissions.

Roll back safely

Rollback should be a tested operation that:

1. stops new candidate assignments;
2. defines what happens to in-flight workflows;
3. routes compatible requests to the stable release;
4. reconciles partial side effects and queued work;
5. retains candidate traces for incident review;
6. confirms stable health after the switch.

If the change includes persistent data or retrieval-index updates, ensure the stable release can still read the state. Forward-only migrations need a separate mitigation plan.

A controlled promotion sequence

offline gates pass
  → internal or shadow observation
  → small eligible canary
  → review hard failures and comparative signals
  → expand one stage
  → repeat until full rollout
  → retain rollback and monitor after promotion

Automate measurement and emergency stops where signals are reliable, but keep a named human release owner. AI Operations covers monitoring and incident response; GitHub Actions for AI applications covers protected deployment workflows.

A canary is successful when it produces trustworthy production evidence while keeping the candidate bounded and reversible. Traffic percentage alone does neither.