How to fix flaky Playwright tests in CI

Tests that pass on your laptop and fail in CI usually share a handful of causes. Here is how to find each one, the code that fixes it, and where Zerocheck fits alongside the suite you keep.

Why this is hard to test

  • •CI runners are slower and have less memory than a developer machine, so timing that was never tight locally becomes tight in CI.
  • •Playwright’s auto-wait checks visibility, enabled state and a stable bounding box. It does not wait for opacity fades, late layout shifts, debounced inputs or client-side route transitions.
  • •Frame locators auto-wait inside embedded documents. Tests still need the correct frame or popup and an assertion on the application outcome.
  • •Parallel workers expose shared test data and ordering dependencies that a single local run never shows.
  • •A passing retry can hide the initial failure when the run does not retain artifacts or traces only the retry.

Approach 1: Find and fix each flake

  1. 1.Set trace: 'on-first-retry' so a retry records DOM snapshots, network and timing without slowing green runs.
  2. 2.Reproduce locally with npx playwright test --repeat-each=50 --retries=0 on the failing file.
  3. 3.Classify the failure: race condition, shared data, environment difference or an external service.
  4. 4.Replace guesses with conditions: expect(locator).toBeVisible(), waitForResponse(), waitForURL(). Never waitForTimeout().
  5. 5.Give each test its own data with unique names and a fresh state; use serial mode only for genuinely dependent steps.
  6. 6.Fix the root cause, then re-enable parallel workers. Retries are a safety net, not a fix.

Approach 2: Where Zerocheck fits alongside your suite

  1. 1.Keep the Playwright tests that earn their maintenance. Pick one repeated manual browser flow the suite does not cover for a separate evaluation.
  2. 2.Describe that flow in Zerocheck’s web app with the same account, fixture and expected outcome, then review the drafted steps and the real browser result.
  3. 3.Break the product on purpose and confirm the failure is explained: failing step, screenshots, console errors, every attempt and the exact test revision.
  4. 4.Move a control and inspect the healing history. A heal is saved only after the unchanged test passes with independent evidence, and a pass after a transient retry stays labeled flaky.

Race conditions that survive auto-wait

Playwright’s auto-wait is good, and its definition of “stable” is precise: the element’s bounding box has not changed across two consecutive animation frames. A slide or a scale moves the box, so Playwright waits for it. What gets through is everything that leaves the box alone. Three kinds of race survive. Fades and late layout: a dialog fading in has its final bounding box from the first frame, so Playwright treats it as ready while it is still transparent and the app may still be ignoring input; content that finishes loading after the check can shift the target. Debounced inputs: a search box that waits after the last keystroke before requesting results will still show stale results when the assertion runs. Client-side navigation: the URL can change before the destination component has fetched its data. The fix is the same shape each time: wait for the condition that means ready, not for time. Wait for the element’s animations to finish, wait for the response the debounce triggers, and assert on content that only exists once the destination has rendered.

import { test, expect } from '@playwright/test';

// Flaky: a fading dialog has a stable box from the first frame, so the click can land while it is still transparent.
test('create project (flaky)', async ({ page }) => {
  await page.goto('/dashboard');
  await page.getByRole('button', { name: 'New project' }).click();
  await page.getByRole('button', { name: 'Create' }).click();
});

// Fixed: wait for the transition to finish before interacting.
test('create project', async ({ page }) => {
  await page.goto('/dashboard');
  await page.getByRole('button', { name: 'New project' }).click();
  const dialog = page.getByRole('dialog');
  await expect(dialog).toBeVisible();
  await dialog.evaluate((el) => Promise.all(el.getAnimations().map((a) => a.finished)));
  await dialog.getByRole('button', { name: 'Create' }).click();
});

// Flaky: the assertion can run before the debounced request has fired.
test('search users (flaky)', async ({ page }) => {
  await page.goto('/users');
  await page.getByPlaceholder('Search').fill('alice');
  await expect(page.getByText('[email protected]')).toBeVisible();
});

// Fixed: register the response wait before typing, then await it.
test('search users', async ({ page }) => {
  await page.goto('/users');
  const results = page.waitForResponse((r) => r.url().includes('/api/users') && r.ok());
  await page.getByPlaceholder('Search').fill('alice');
  await results;
  await expect(page.getByText('[email protected]')).toBeVisible();
});

Iframes and popups load on their own schedule

Playwright frame locators resolve the embedded document and return ordinary locators: fill and click retain their actionability waiting inside cross-origin frames. A later-rendered input does not, by itself, require a separate wait. Use the correct frame and a unique locator. An explicit visibility assertion can make a failure easier to diagnose and give field rendering its own bounded deadline. It does not replace actionability checks or prove the embedded application has finished its business operation. Verify the expected result after the action. Popups add one more lifecycle. Some providers open a new page rather than an iframe. Register page.waitForEvent('popup') before the click that opens it, then wait for the popup’s own load state before interacting.

import { test, expect } from '@playwright/test';

// Locator.fill already auto-waits inside the frame.
// The assertion below adds a named rendering checkpoint and its own timeout.
test('enter card details', async ({ page }) => {
  await page.goto('/checkout');
  const stripe = page.frameLocator('iframe[title="Secure payment input frame"]');
  const card = stripe.getByPlaceholder('1234 1234 1234 1234');
  await expect(card).toBeVisible({ timeout: 10_000 });
  await card.fill('4242424242424242');
});

// Popups: listen before the click, then wait for the popup's own load state.
test('sign in with a popup provider', async ({ page }) => {
  await page.goto('/login');
  const popupPromise = page.waitForEvent('popup');
  await page.getByRole('button', { name: 'Sign in with Google' }).click();
  const popup = await popupPromise;
  await popup.waitForLoadState('domcontentloaded');
  await popup.getByLabel('Email').fill(process.env.TEST_EMAIL!);
  await popup.getByRole('button', { name: 'Next' }).click();
  await expect(page.getByText('Welcome back')).toBeVisible({ timeout: 15_000 });
});

Shared data and ordering dependencies

Every test passes alone. Every test passes in a fixed order. Run the suite in parallel and a few fail at random. The cause is shared mutable state: two tests that create the same user, or one that empties a list another one counts. Playwright runs files in parallel by default, so order across files is not deterministic. The fix is isolation. Each test creates its own data with a unique name and does not depend on what another test left behind. Where steps are genuinely sequential, put them in one describe block in serial mode and share a page explicitly, so the dependency is visible in the code rather than hidden in worker scheduling. Authentication is the shared state most teams get wrong. Sign in once in a setup project, save the storage state to a file, and let every test start from that file. Tests then share a saved session without sharing a live browser. Share an account only when tests do not mutate conflicting server-side state; otherwise use separate accounts and fixtures. Add .auth/ to .gitignore and exclude saved authentication state from shared artifacts because it contains reusable login credentials.

import { test, expect, type Page } from '@playwright/test';

// Flaky in parallel: two workers both create 'My Project' and one hits a uniqueness error.
test('create project (flaky)', async ({ page }) => {
  await page.goto('/projects');
  await page.getByRole('button', { name: 'New' }).click();
  await page.getByLabel('Name').fill('My Project');
  await page.getByRole('button', { name: 'Create' }).click();
});

// Fixed: every test owns its data.
test('create project', async ({ page }, testInfo) => {
  const name = `Project ${testInfo.workerIndex}-${Date.now()}`;
  await page.goto('/projects');
  await page.getByRole('button', { name: 'New' }).click();
  await page.getByLabel('Name').fill(name);
  await page.getByRole('button', { name: 'Create' }).click();
  await expect(page.getByText(name)).toBeVisible();
});

// Genuinely dependent steps: one worker, one shared page, in order.
test.describe('checkout', () => {
  test.describe.configure({ mode: 'serial' });
  let page: Page;
  test.beforeAll(async ({ browser }) => { page = await browser.newPage(); });
  test.afterAll(async () => { await page.close(); });

  test('adds an item', async () => {
    await page.goto('/products/widget');
    await page.getByRole('button', { name: 'Add to cart' }).click();
    await expect(page.getByTestId('cart-count')).toHaveText('1');
  });

  test('checks out', async () => {
    await page.goto('/cart');
    await page.getByRole('button', { name: 'Checkout' }).click();
    await expect(page).toHaveURL(/\/confirmation/);
  });
});

CI is a different machine

A CI runner is slower than your laptop, has less memory, and may render fonts differently under headless Chromium. Container memory limits can kill the browser mid-test, which surfaces as a cryptic “target closed” error rather than an out-of-memory message. Timezone and locale differ unless you pin them, so a date assertion written in one zone fails in another. The configuration below is tuned for CI stability rather than speed. Timeouts are longer, workers start at one so data collisions disappear while you fix them, retries are on in CI only so a flake is felt locally, and traces are captured on first retry. The setup project signs in once and saves the storage state that every test starts from.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  timeout: 60_000,
  expect: { timeout: 10_000 },
  // Retries unblock the team; they do not fix anything. Keep 0 locally so flakes hurt.
  retries: process.env.CI ? 2 : 0,
  // One worker in CI removes data collisions. Raise it once the shared-data fixes land.
  workers: process.env.CI ? 1 : undefined,
  reporter: process.env.CI ? [['github'], ['html', { open: 'never' }]] : 'list',
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
    timezoneId: 'UTC',
    locale: 'en-US',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    navigationTimeout: 30_000,
    actionTimeout: 15_000,
  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'], storageState: '.auth/user.json' },
      dependencies: ['setup'],
    },
  ],
  webServer: {
    command: 'npm run start',
    port: 3000,
    timeout: 120_000,
    reuseExistingServer: !process.env.CI,
  },
});

// Add .auth/ to .gitignore; never upload it as a shared artifact.
// Share this account only across tests without conflicting server-side writes.
// auth.setup.ts: sign in once, save the session for every test.
import { test as setup } from '@playwright/test';

setup('authenticate', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill(process.env.CI_TEST_EMAIL!);
  await page.getByLabel('Password').fill(process.env.CI_TEST_PASSWORD!);
  await page.getByRole('button', { name: 'Log in' }).click();
  await page.waitForURL('/dashboard');
  await page.context().storageState({ path: '.auth/user.json' });
});

How Zerocheck exposes recovery

A moved control can be resolved again from its description. The result records old and new targets, the reason and validation state. A repair is saved only after the unchanged test passes with independent evidence; a failed assertion stays a failure. A disabled intended control fails rather than healing to a different enabled control.

A changing frame does not justify another purchase

A real evaluation reproduced a frame changing before the first action. Zerocheck now retries incomplete page observations in the same browser with a bounded read budget. Controlled fixtures verify that an error after a dispatched click still produces one click and a failed result. This verifies recovery behavior; it is not proof that a payment succeeded or that a selector healed.

Read the trace instead of guessing

When a test flakes only in CI, the trace viewer is the tool that turns “works on my machine” into a specific fix. A trace records every action with timing, a DOM snapshot at each step, network requests and console output. With trace: 'on-first-retry' the trace describes the retry, not the initial failing attempt. Use trace: 'retain-on-failure' when you need the original failure's trace, accepting its recording overhead. Treat traces as sensitive test artifacts. Get the trace off the runner by uploading the test-results directory even when a retry makes the job pass, then open it locally with npx playwright show-trace. Step through to the failing action and read the timing: a click that fired before an animation ended, or a request that took far longer in CI than locally. That is the evidence a fix can be written against.

# .github/workflows/e2e.yml
name: E2E
on: [pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test
        env:
          CI: true
      # Keep available evidence even when a retry passes.
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-results
          path: test-results/
          if-no-files-found: ignore
          retention-days: 7

# Locally: npx playwright show-trace test-results/<test-name>/trace.zip

Common pitfalls

  • —Do not reach for retries first. A test that needs retries to pass will fail at the worst possible moment.
  • —Do not fix timing with page.waitForTimeout(). It is slow and still a guess. Wait for the condition instead.
  • —Do not leave workers at one permanently. It hides data collisions and multiplies CI time. Fix the shared data, then raise it.
  • —Do not ignore the first flake. It is the easiest one to diagnose; after twenty, the team has learned to ignore red builds.
  • —Do not blame Playwright by default. Most flakes come from test design or application behavior, not the framework.

FAQ

Why do Playwright tests pass locally but fail in CI?

Three usual reasons. CI runners have less CPU and memory, so operations take longer and timeout-sensitive assertions fail. Headless rendering and fonts can differ, so layout-dependent locators miss. Network latency to external services is higher and more variable than on localhost.

Should I use retries to handle flaky tests?

As a safety net, not a fix. Retries in CI keep the team unblocked while you investigate, and every retried pass is a flake you have chosen to tolerate. Track the retry rate and fix the causes behind it.

How do I debug a Playwright test that only flakes in CI?

Set trace: 'on-first-retry', upload the test-results directory as a CI artifact even when the retry passes, then open the trace with npx playwright show-trace. It records the first retry, not the original failure. Use trace: 'retain-on-failure' when you need the failed attempt itself.

Does Playwright detect flaky tests?

It reports a test as flaky when it fails and then passes on retry, and it does not keep history across runs. Zerocheck records every attempt with screenshots, console errors and the exact test revision, and a pass after a transient retry stays labeled flaky rather than being hidden.

When should I delete a flaky test?

When you have investigated it properly and cannot make it deterministic, ask whether it covers a behavior the release depends on. If it does, rewrite it with isolation and explicit waits. If not, delete it. A test nobody trusts trains the team to ignore CI.

How to fix flaky Playwright tests in CI

Describe a customer flow or paste your release checklist, review the browser result, and save the tests you want to rerun.

Get a demo