βš™οΈ AI Operations
Β· 3 min read
Last updated on

Playwright Timeout Errors in AI Browser Agents: Diagnose and Fix Them


Test timeout of 30000ms exceeded

In an AI browser workflow, this message rarely tells you what actually stalled. The model request may still be streaming, an agent may be waiting for approval, a selector may target generated text, or a retry may have created unexpected state.

Do not begin by making every timeout larger. Identify which boundary expired and which state the application had reached.

Find the operation that timed out

Playwright distinguishes test, action, assertion and navigation timeouts. Capture the error, previous application logs and a trace:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  trace: 'on-first-retry',
  screenshot: 'only-on-failure',
  retries: process.env.CI ? 1 : 0,
});

Open the recorded trace:

npx playwright show-trace test-results/**/trace.zip

Inspect the final successful action, DOM snapshot, console messages and network requests. A timeout at click() may mean an overlay blocked the action; a timeout at an assertion may mean the backend job never changed state.

Wait for application state, not elapsed time

Avoid fixed sleeps:

// Fragile
await page.waitForTimeout(15_000);

// Better: wait for a product state
await expect(page.getByTestId('run-status')).toHaveText('Completed', {
  timeout: 90_000,
});

AI workflows should expose explicit states such as queued, running, awaiting_approval, completed, failed and cancelled. Those states make the product observable and the test deterministic.

Do not use networkidle as a universal completion signal. Streaming, polling and telemetry may keep network activity alive after the user-visible result is ready.

Long model responses and streaming

Test the streaming lifecycle rather than waiting only for the final paragraph:

await page.getByRole('button', { name: 'Generate' }).click();
await expect(page.getByTestId('stream-status')).toHaveText('Streaming');
await expect(page.getByTestId('stop-generation')).toBeEnabled();
await expect(page.getByTestId('stream-status')).toHaveText('Completed', {
  timeout: 60_000,
});

For the main browser suite, mock a representative sequence of stream events. Run a smaller real-provider suite separately with a larger, evidence-based budget.

Agent actions and approval waits

An agent may pause intentionally before a consequential tool call. A test that waits directly for the final effect will look hung.

await expect(page.getByText('Approval required')).toBeVisible();
await page.getByRole('button', { name: 'Approve action' }).click();
await expect(page.getByText('Action completed')).toBeVisible();

Assert every safety boundary. If the product expects a human decision, the test must provide it rather than extending the timeout.

Retries can hide state bugs

Retries are useful for collecting a trace, but they can repeat side effects. Use unique test data and idempotent APIs. Before enabling retries, ask:

  • Did the first attempt create a job even though the UI timed out?
  • Does the retry reuse a conversation, task or browser session?
  • Can the same tool action execute twice?
  • Does cleanup run after a failed test?

A passing retry is still evidence of instability. Track flaky tests rather than treating them as healthy.

Selector failures in generated interfaces

Do not select an element by exact model-generated wording. Prefer stable roles, accessible names for fixed controls, and test IDs for stateful components:

await page.getByRole('button', { name: 'Approve action' }).click();
await expect(page.getByTestId('tool-result')).toBeVisible();

If a model controls which tools or cards appear, stub the model decision for deterministic UI tests and evaluate decision quality separately.

Raise a timeout only at the correct boundary

When a real workflow legitimately needs more time, scope the increase:

test('completes a queued evaluation run', async ({ page }) => {
  test.setTimeout(120_000);
  await page.goto('/evaluations');
  await page.getByRole('button', { name: 'Run evaluation' }).click();
  await expect(page.getByTestId('evaluation-status')).toHaveText('Completed', {
    timeout: 90_000,
  });
});

Keep action timeouts shorter so a blocked click fails quickly. Document why a long test needs its budget.

CI-only timeout failures

If the test passes locally but fails in CI, compare:

  • CPU and memory available to browser workers;
  • worker count and parallel model requests;
  • provider rate limits;
  • browser and dependency versions;
  • missing secrets or environment configuration;
  • network access and DNS;
  • whether traces and videos exhaust disk space.

Reduce workers as a diagnostic, not an automatic permanent fix:

export default defineConfig({
  workers: process.env.CI ? 2 : undefined,
});

A reliable debugging order

  1. Reproduce with one worker and trace enabled.
  2. Identify the exact expired assertion or action.
  3. Inspect backend job and provider request IDs.
  4. Replace fixed sleeps with explicit application state.
  5. Mock the model in deterministic E2E tests.
  6. Verify approval and retry semantics.
  7. Increase only the timeout justified by measured behavior.

Place this within Playwright vs Cypress for AI applications, the AI Testing & Evaluation hub, AI Operations and the Coding Agents hub.

Primary documentation