Deploying AI agents to production is not like deploying regular software. Agents are non-deterministic, can consume unlimited resources, and fail in unexpected ways. You need a different deployment process.
Here’s the checklist that covers everything.
Pre-deployment
Testing
- Run agent on test suite (50+ tasks minimum)
- Success rate > 80% for target task category
- Cost per task within budget (test 10+ runs)
- Latency within acceptable range (p95 < target)
- No safety violations in adversarial tests
- Consistency check (same input, similar output across 5 runs)
Security
- Agent runs in sandboxed container
- Non-root user in container
- Read-only filesystem (except output directory)
- Network restricted to required services only
- Credentials injected via secrets manager (not hardcoded)
- No PII in agent prompts or outputs (verify)
- Output validation rules defined
Cost controls
- Per-task cost limit set (default: $5)
- Per-agent daily limit set (default: $50)
- Per-team monthly limit set (default: $500)
- Budget alerts configured (50%, 80%, 100%)
- Cost monitoring dashboard created
Infrastructure
- Container image built and tested
- Resource limits set (CPU, memory, disk)
- Health checks configured
- Logging pipeline configured
- Monitoring dashboards created
- Alert rules defined
Deployment
Rollout strategy
- Start with canary deployment (5% of traffic)
- Monitor for 24 hours
- Gradually increase to 25%, 50%, 100%
- Rollback plan documented and tested
Configuration
- Environment variables set (not hardcoded)
- Feature flags for agent capabilities
- Model version pinned (don’t auto-update)
- Rate limits configured per provider
Monitoring
- Success rate tracked per agent
- Cost per task tracked
- Latency tracked (p50, p95, p99)
- Error rate tracked by type
- Token usage tracked per model
Post-deployment
First 24 hours
- Monitor success rate (alert if < 70%)
- Monitor cost (alert if > 2x expected)
- Monitor error rate (alert if > 10%)
- Review sample outputs manually
- Check for infinite loops or hung agents
First week
- Review cost trends
- Analyze failure patterns
- Adjust cost limits if needed
- Update test suite with real-world cases
- Document any incidents
Ongoing
- Weekly cost review
- Monthly success rate review
- Quarterly governance policy review
- Annual security audit
Rollback plan
Automatic rollback triggers
- Success rate drops below 50%
- Cost exceeds 3x daily limit
- Error rate exceeds 20%
- Agent hangs for > 30 minutes
Manual rollback steps
- Disable agent via feature flag
- Redirect traffic to previous version
- Investigate root cause
- Fix and re-test
- Re-deploy with canary
Incident response
Severity levels
| Level | Description | Response Time | Example |
|---|---|---|---|
| P1 | Agent causing damage | 15 minutes | Data deletion, credential leak |
| P2 | Agent completely failed | 1 hour | 0% success rate |
| P3 | Agent degraded | 4 hours | 50% success rate |
| P4 | Minor issue | 24 hours | Increased cost, slow latency |
Response steps
P1 (immediate):
- Kill all agent instances
- Disable agent via feature flag
- Assess damage scope
- Notify stakeholders
- Begin root cause analysis
P2-P4:
- Investigate root cause
- Implement fix
- Test fix
- Deploy fix
- Monitor recovery
Environment-specific considerations
Development
- Use mock APIs for testing
- Set low cost limits ($1/task)
- Enable verbose logging
- Allow file system write access
Staging
- Use real APIs with test data
- Set moderate cost limits ($5/task)
- Enable standard logging
- Restrict file system access
Production
- Use real APIs with real data
- Set production cost limits
- Enable minimal logging (errors + metrics)
- Full sandboxing
My take
The most important items on this checklist are:
- Cost limits: Per-task and per-agent. Prevent runaway spending.
- Sandboxing: Every agent in its own container. Prevent cross-contamination.
- Monitoring: Success rate, cost, latency. Know when things go wrong.
- Rollback plan: Know how to undo a bad deployment.
Don’t try to check every box on day one. Start with cost limits and sandboxing. Add monitoring and governance as you scale.
The biggest deployment mistake is treating agents like regular software. They’re not. They’re non-deterministic, resource-hungry, and fail in unexpected ways. Plan accordingly.
FAQ
How is deploying agents different from regular software?
Agents are non-deterministic (different output each run), resource-hungry (can consume unlimited tokens), and fail in unexpected ways (hallucinations, infinite loops). Traditional deployment practices don’t cover these risks.
What’s the minimum deployment checklist?
Cost limits, sandboxing, and monitoring. These three items handle most production risks.
How do I handle agent failures in production?
Set automatic rollback triggers (success rate < 50%, cost > 3x limit). When triggered, disable the agent and investigate. Don’t let failing agents run unattended.
Should I use canary deployments for agents?
Yes. Start with 5% of traffic, monitor for 24 hours, then gradually increase. Agents can behave differently in production than in testing.
How often should I update agent models?
Pin model versions. Don’t auto-update. Test new versions in staging before deploying to production. Model updates can change agent behavior unexpectedly.