Image generation, document ingestion, fine-tuning, and long-running agents often outlive an HTTP request. The application accepts work, returns a job ID, and later receives or sends events such as job.completed, job.failed, or agent.approval_required.

Webhooks make those workflows composable. They also introduce duplicates, reordering, forged requests, delayed delivery, and sensitive payloads. A production design assumes all five will happen.

Reference flow

client → create job → durable queue → AI worker
                     ↓                 ↓
                 job state ← result/event
                     ↓
              webhook outbox → subscriber

The HTTP request should not be the only record of the job. Persist state before acknowledging accepted work.

Use an event envelope

{
  "id": "evt_01J...",
  "type": "generation.completed",
  "createdAt": "2026-08-24T10:00:00Z",
  "attempt": 1,
  "data": {
    "jobId": "job_01J...",
    "status": "completed",
    "resultUrl": "https://example.com/results/job_01J..."
  }
}

Keep the event ID stable across retries. The delivery attempt changes; the business event does not. Version the envelope or event type when consumers need a breaking schema change.

At-least-once delivery is the practical default

If a receiver processes an event but its 2xx response is lost, the sender cannot know whether processing succeeded. Retrying may deliver a duplicate.

Design for at-least-once delivery:

• the sender retries transient failures;
• the receiver stores processed event IDs;
• business operations use idempotency constraints;
• duplicates return success without repeating the side effect.

Read Idempotency for AI Agents and Reliable AI Workflows.

Sign every remote webhook

Use an HMAC or asymmetric signature over the raw request body and a timestamp. Verify the signature before parsing or processing.

import crypto from 'node:crypto';

function validSignature(rawBody, timestamp, received, secret) {
  const age = Math.abs(Date.now() / 1000 - Number(timestamp));
  if (!Number.isFinite(age) || age > 300) return false;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${timestamp}.${rawBody}`)
    .digest('hex');

  const a = Buffer.from(expected);
  const b = Buffer.from(received || '');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

Use the raw bytes exactly as received. Store webhook secrets in an approved secret system, rotate them with an overlap window, and distinguish test from production endpoints. See AI Security & Credentials.

Acknowledge quickly, process asynchronously

The receiver should authenticate, validate the envelope, persist it or enqueue it, and return quickly. Calling another model before returning increases timeout and duplicate-delivery risk.

receive → verify → deduplicate → enqueue → return 202
                                      ↓
                               process in worker

If queueing fails, do not return success. A 2xx should mean the event is durably accepted—not merely that the server parsed JSON.

Retry policy

Retry connection failures, timeouts, 429, and most 5xx responses with exponential backoff and jitter. Do not blindly retry permanent 4xx failures such as an invalid endpoint or rejected signature.

A practical delivery record contains:

• event ID;
• endpoint ID;
• attempt number;
• scheduled time;
• response status;
• bounded response excerpt;
• final state;
• next retry time.

Set a maximum delivery age and attempt count. After that, move the delivery to a dead-letter state that an operator can inspect and replay.

Ordering and concurrency

Webhooks can arrive out of order. job.completed may be processed before a delayed job.started event. Include event timestamps or sequence numbers, but do not assume clocks alone establish a perfect order.

Use a state machine that rejects invalid backwards transitions:

queued → running → completed
                 ↘ failed
queued → cancelled

Serialize processing per job when concurrent events can conflict. Global ordering is expensive and rarely necessary.

Keep payloads small and privacy-aware

AI results may contain personal data, source documents, prompts, or generated media. Prefer stable identifiers and short-lived authenticated result URLs over embedding entire results in every delivery.

Never put provider keys, internal credentials, or unrestricted storage URLs in an event. Define retention for event payloads and delivery logs.

Agent-specific events

Useful agent events include:

• agent.run.started;
• agent.tool.requested;
• agent.approval_required;
• agent.run.completed;
• agent.run.failed;
• agent.budget_exceeded.

Do not use a webhook as authorization for a destructive action unless the event is authenticated, the action is allowed by current policy, and replay cannot repeat it. Human approval tokens should be narrow, expiring, and single-use.

Outbox pattern

When changing database state and emitting an event must succeed together, write the business change and an outbox record in one database transaction. A separate dispatcher sends the event and marks delivery progress.

This avoids the gap where the job completes but the process crashes before publishing its webhook.

Observability

Measure:

• delivery success and final failure rate;
• end-to-end event latency;
• attempts per event;
• signature failures;
• duplicate rate;
• queue age;
• dead-letter volume;
• replay outcomes.

Correlate the event ID, job ID, and originating request ID without logging sensitive content.

Production checklist

• Persist job state before acknowledging work.
• Sign remote events and reject stale timestamps.
• Deduplicate by stable event ID.
• Make downstream side effects idempotent.
• Return success only after durable acceptance.
• Back off transient retries and dead-letter exhausted delivery.
• Model ordering through valid state transitions.
• Keep sensitive results out of routine payloads.
• Provide controlled inspection and replay tools.

Place this pattern within AI Application Architecture, connect failure policy through Handling AI API Failures, and enforce access through Authentication for AI Applications.