A production model decision is a system decision. The best candidate is not necessarily the model with the highest public benchmark score; it is the configuration that meets your application’s quality, reliability, latency, cost and safety requirements on representative work.

Use this checklist before a new deployment, provider migration or material model update. Build the underlying evidence with the AI Testing & Evaluation hub and a versioned LLM evaluation dataset.

1. Define the job and release threshold

• What user outcome must the model support?
• Which tasks are in scope and explicitly out of scope?
• Which errors are tolerable, recoverable or release-blocking?
• Which user, language and risk slices need separate results?
• What baseline must the candidate equal or improve?

Do this before reviewing outputs. Otherwise the team will move the success criteria toward whichever model demos best.

2. Measure task quality

Choose task-specific evidence:

• exact or constrained labels for classification;
• executable tests for generated code;
• required facts and citations for grounded answers;
• schema and semantic checks for tool arguments;
• calibrated rubrics for subjective quality;
• final state and side effects for agents.

Report per-case and per-slice results, not only an overall mean. An improvement on easy requests can conceal a decline in the workflow that generates revenue or creates risk.

3. Test hallucinations and unsupported claims

Include cases where:

• the supplied context contains the answer;
• evidence is missing and the correct behavior is uncertainty;
• sources conflict;
• the question contains a false premise;
• retrieved documents are irrelevant or stale;
• a tool returns incomplete data.

Separate retrieval failure from generation failure. For RAG, first verify that the required source was retrieved, then evaluate whether the answer used it faithfully.

4. Validate instructions, schemas and tools

Test whether the candidate:

• follows priority and formatting constraints;
• produces valid structured output;
• selects only allowed tools;
• generates schema-valid arguments;
• preserves approval and permission boundaries;
• recovers safely from tool errors;
• avoids duplicate side effects after retries.

Provider-constrained output does not replace business validation. Link schema checks to Structured Outputs and malformed-data handling to the JSON parsing foundation.

5. Measure reliability, not one good run

Variable systems need repeated evidence where variance matters. Track:

• success rate and failure distribution;
• consistency of task outcome—not identical wording;
• invalid-output and refusal rates;
• retry and fallback frequency;
• provider errors and timeouts;
• performance across context lengths and concurrent load;
• severe failures as hard counts.

Use a pinned or recorded model configuration where possible. A moving provider alias can change behavior without an application-code diff.

6. Benchmark latency end to end

Measure the experience the user receives:

• time to first usable output;
• total completion time;
• streaming stalls;
• tool and retrieval latency;
• median and tail latency under realistic concurrency;
• timeout and cancellation behavior.

Do not benchmark only a direct provider call when production also includes authentication, retrieval, gateways, queues and tools. AI Application Architecture defines those boundaries.

7. Model total cost

Measure cost per successful task, not merely list price per token. Include:

• input, output and reasoning usage where billed;
• retrieval and embedding calls;
• tool and external API costs;
• retries, fallbacks and failed attempts;
• caching behavior;
• evaluation and observability overhead;
• infrastructure for self-hosted models.

Report median and tail cost for real task slices. A cheap model can be expensive if it produces longer outputs, retries often or requires a second model to repair results.

8. Evaluate safety and security

Run product-specific cases for:

• prompt injection and untrusted retrieved instructions;
• sensitive-data disclosure;
• disallowed or harmful content;
• tenant isolation;
• overbroad tool permissions;
• secret handling and logging;
• unsafe actions disguised as ordinary requests;
• over-refusal on legitimate tasks.

Safety is not one generic score. Define the threat, expected response and severity. The AI Security foundation covers identity, secrets and agent permissions.

9. Check privacy and operational constraints

Before deployment, confirm:

• what inputs and outputs the provider stores;
• whether data may be used for training;
• retention and deletion controls;
• processing regions and contractual requirements;
• logging and abuse-monitoring behavior;
• support for least-privilege credentials;
• incident, deprecation and availability processes.

Verify current provider documentation and contracts rather than copying an old comparison table.

10. Test fallback and migration paths

A fallback is only real if it has been exercised. Verify:

• schema and tool compatibility;
• prompt and safety behavior;
• quality on critical tasks;
• latency and cost under fallback conditions;
• state consistency during a mid-workflow failure;
• observability that distinguishes primary and fallback runs.

Do not silently route high-risk work to a weaker model merely to preserve uptime.

11. Prevent regressions in CI/CD

Record the model, prompt, dataset, grader, retrieval corpus and code versions. Run affected slices on pull requests and the broader suite before protected releases. Save per-case evidence and require human review for high-impact changes.

Use LLM regression testing for the comparison pipeline and GitHub Actions for AI applications for reviewable gates and artifacts.

12. Plan the production rollout

Offline evaluation is necessary but incomplete. Release progressively where risk permits:

1. internal or shadow traffic;
2. a limited canary;
3. monitored expansion;
4. rollback when predefined signals fail;
5. reviewed production samples feeding new regression cases.

Connect quality, cost and reliability signals to AI Operations. Ownership matters: someone must have authority to stop or roll back the release.

Deployment decision record

End the evaluation with a short decision record:

Candidate and baseline:
Dataset and grader versions:
Critical slices tested:
Quality result:
Latency and cost result:
Safety and reliability result:
Known limitations:
Fallback status:
Rollout and rollback plan:
Approver:

The checklist is complete when it supports a defensible deployment decision—not when every box has a reassuring answer.