Picking an AI model for production is not the same as picking one for a weekend prototype. The model that demos well on a blog post might fall apart on your actual data, blow through your budget in a week, or land you in a compliance nightmare.

Before you sign up for an API or commit engineering time to an integration, run through these 15 questions. They are organized into five categories: quality, cost, latency, privacy, and vendor risk. Print this out, pin it to your wall, or paste it into your evaluation spreadsheet — whatever keeps you honest.

If you want a deeper dive on evaluation methodology, start with our guide on how to evaluate AI models for your use case.

Quality

Quality is the obvious starting point, but most teams stop at “it gave a good answer once.” That is not evaluation. That is a demo.

1. Does the model handle your edge cases?

Every domain has them. Legal text with nested clauses. Medical abbreviations. Multilingual customer support tickets with code-switching. Build a test set of 50–100 examples that represent your hardest inputs and run them through every candidate model. If a model fails on the edges, it will fail in production — that is where your users will notice.

2. What is the hallucination rate on your data?

Benchmark hallucination rates from leaderboards are a starting point, not an answer. Measure hallucination against your own corpus. Feed the model questions where the correct answer is “I don’t know” and see how often it fabricates something confident. For retrieval-augmented setups, check whether the model sticks to the provided context or invents details.

3. How well does it follow complex instructions?

Simple prompts are easy. The real test is multi-step instructions with constraints: “Summarize this in three bullet points, each under 20 words, in formal tone, without mentioning competitor names.” If the model ignores one constraint out of four, that is a 25% instruction-following failure rate — and your users will feel it.

For side-by-side comparisons across models, see our AI model comparison page.

Cost

The cheapest model is not always the most cost-effective. Token efficiency, caching, and output verbosity all affect your real bill.

4. What is the per-token pricing — input and output separately?

Most providers charge differently for input and output tokens. A model that is cheap on input but expensive on output will hurt if your use case generates long responses. Map your expected input/output ratio and calculate accordingly.

5. How token-efficient is the model for your task?

Two models can produce the same quality answer, but one might use 40% fewer tokens to do it. A concise model saves money on every single request. Run your test set and compare total token counts at equivalent quality levels.

6. Does the provider support prompt caching?

If you send the same system prompt or context prefix with every request, prompt caching can cut input costs dramatically — sometimes by 50–90%. Check whether your provider offers it, how it works, and whether your architecture can take advantage of it.

Latency

Users notice latency. Internal tools tolerate more of it. Know your requirements before you benchmark.

7. What is the time to first token (TTFT)?

For interactive applications — chatbots, copilots, search — TTFT matters more than total generation time. A model that starts streaming in 200ms feels fast even if the full response takes 5 seconds. Measure TTFT under realistic load, not just on an idle endpoint.

8. What is the sustained throughput?

Throughput is tokens per second once generation starts. A model with great TTFT but slow throughput will frustrate users on long outputs. Test with response lengths that match your production use case.

9. Does the model support streaming, and does your stack handle it?

Streaming is table stakes for user-facing applications. But it is not just about the model — your API gateway, backend framework, and frontend all need to handle streamed responses. Verify the full chain, not just the model endpoint.

Privacy

This is where deals die and legal teams get involved. Get clear answers before you write a single line of integration code.

10. Where does your data go when you send it to the API?

Understand the data flow. Is the data processed in a specific region? Is it stored temporarily for abuse monitoring? Can it end up in a training dataset? Read the data processing agreement, not just the marketing page.

11. Is the provider GDPR-compliant (and compliant with your other regulatory requirements)?

GDPR is the baseline for European users, but you may also need SOC 2, HIPAA, or industry-specific certifications. Ask for documentation, not promises. Check whether the provider has a Data Processing Agreement (DPA) ready to sign.

We maintain a list of which AI APIs are GDPR-compliant — start there if you are serving European users.

12. What is the data retention policy?

Some providers retain inputs and outputs for 30 days. Others delete on completion. Some let you configure retention. Know exactly how long your data lives on someone else’s servers and whether that aligns with your own data governance policies.

For a detailed breakdown, see our AI data retention policies comparison.

Vendor Risk

Models get deprecated. APIs change. Companies pivot. Plan for it.

13. How locked in will you be?

Evaluate how much of your code is provider-specific. Proprietary SDKs, custom fine-tuning formats, and provider-specific function calling schemas all increase switching costs. Use abstraction layers where practical, and keep your prompts portable.

14. What is the model deprecation policy?

Providers retire model versions. Sometimes with 6 months notice, sometimes with 6 weeks. Know the policy before you depend on a specific model version. If the provider has a history of breaking changes, factor that maintenance cost into your decision.

Read about how AI model version changes affect production systems to understand what is at stake.

15. Do you have a fallback option?

If your primary model goes down, gets deprecated, or doubles in price overnight, what do you do? Identify at least one fallback model that meets your minimum quality bar. Test it periodically. The best time to find a backup is before you need one.

How to Use This Checklist

Don’t try to answer all 15 questions in an afternoon. Here is a practical approach:

1. Filter on quality first. Run your edge-case test set. Eliminate models that fail your minimum bar.
2. Model the cost. Estimate monthly spend based on your projected volume. Include token efficiency differences.
3. Benchmark latency under load. Not on a quiet Sunday morning — under conditions that resemble your production traffic.
4. Get legal sign-off on privacy. Do this early. A model that fails compliance is not a candidate, no matter how good it is.
5. Assess vendor risk honestly. The best model from the riskiest vendor might not be the right choice.

The goal is not to find the perfect model. It is to find the model that is good enough across all five dimensions — and to know exactly where the tradeoffs are before your users find them for you.