AI applications need ordinary software checks plus controls for prompts, model behaviour, datasets, credentials and cost. A pipeline that only runs unit tests can ship code successfully while degrading answer quality or exposing a provider key.

This guide builds the delivery path around those risks. Pair it with Git workflows for coding agents and the AI operations hub.

1. Scope pull-request checks

Start with deterministic checks on every pull request:

name: pull-request
on:
  pull_request:

permissions:
  contents: read

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run typecheck
      - run: npm test
      - run: npm run build

Pin runtime versions and use lockfile installs. Give the workflow read-only permissions unless a job genuinely needs more.

2. Separate model evaluations

Model evaluations may be slower, nondeterministic and billable. Run a small stable gate on pull requests and broader evaluation suites on schedule or before release.

Store dataset versions and evaluation code with the repository. Record model, prompt and configuration versions in results. A score without those inputs cannot explain a regression.

Set a budget and concurrency limit so a faulty workflow cannot create unlimited API spend.

3. Test structured outputs and tools

Add contract tests for:

• schema-valid model responses;
• missing and extra tool arguments;
• refused or unavailable tools;
• retry and idempotency behaviour;
• authorization before side effects;
• fallback models and degraded states.

Mock providers for deterministic unit tests, then run a small number of live integration tests against approved non-production accounts.

4. Protect secrets

Use environment-scoped secrets and short-lived cloud identity where possible. Do not expose production keys to workflows triggered from forks.

permissions:
  contents: read
  id-token: write

The id-token permission should only exist on the deployment job that exchanges it for a short-lived credential. See AI security and environment-variable management.

5. Scan agent-generated changes

Coding agents can accidentally add .env files, generated databases or sensitive logs. CI should check repository policy, not trust the agent report:

• secret scanning;
• dependency review;
• licence policy;
• forbidden generated artefacts;
• unexpected large files;
• changed migrations and infrastructure plans.

Require a human reviewer for authentication, billing, destructive operations and production infrastructure.

6. Build immutable artefacts

Build the application or container once, identify it with the commit SHA and promote the same artefact between environments. Rebuilding for production can silently change dependencies.

For containers:

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: registry.example.com/ai-app:${{ github.sha }}

Scan the resulting image and generate provenance where the environment requires it.

7. Validate database changes

Run migrations against an ephemeral or staging database before deployment. Check backward compatibility when old and new application versions overlap.

Embedding indexes and model caches are derived state; product records and audit logs are not. The deployment plan must distinguish rebuildable data from state that requires backup and migration.

8. Use protected deployment environments

GitHub environments can separate staging and production secrets and require approval:

deploy-production:
  environment: production
  needs: [verify, evaluate]
  runs-on: ubuntu-latest

An approval should show the artefact, evaluation change, migration plan and rollback path—not merely a green check.

9. Verify after deployment

Smoke-test the running system:

• health and readiness endpoints;
• one safe model request;
• streaming completion;
• tool calls blocked without permission;
• queue and worker processing;
• representative page and API responses.

Monitor errors, first-token latency, fallback rate and cost after release. A successful upload is not a successful deployment.

10. Make rollback explicit

Record the previous artefact and provide a bounded rollback job. Some migrations and external side effects cannot be reversed automatically; state that before deployment.

Cancel superseded workflow runs and use concurrency controls:

concurrency:
  group: production
  cancel-in-progress: false

AI CI/CD works when deterministic software checks, model evaluations and human risk decisions form one auditable path. Automation should make releases repeatable without pretending uncertain model behaviour has become deterministic.