Why are my playwright tests sometimes successful and sometimes not?

Question

I have some playwright tests which are successful sometimes and sometimes not.

(I actually had this problem with a website which is still in development. But I adapted it to a public url.)

These are my files (file content is at the bottom of this post):

├── Dockerfile
├── package.json
├── tests
│   ├── 00.spec.ts
│   ├── 01.spec.ts
│   ├── 02.spec.ts
│   ├── 03.spec.ts
│   ├── 04.spec.ts

tests/00.spec.ts combines the tests 01-04 in one file. So I have four different tests and each is run twice (once in 00.spec.ts and once in 0x.spec.ts).

Execution command:

docker build -t playwright-test . && docker run --rm -it playwright-test

I ran the execution command ten times with the following results:

1. all successful

2. 04 failed, others were successful

3. all successful

4. 00 (league form 1) failed, others were successful

5. 04 failed, others were successful

6. 03 failed, others were successful

7. 02 failed, others were successful

8. 00 (league form 1) failed, others were successful

9. all successful

10. all successful

When a test fails, I see this error:

  1) tests/00.spec.ts:54:7 › league form 1 ────────────────────────────────────────────────────────

    Test timeout of 30000ms exceeded.

    Error: locator.click: Test timeout of 30000ms exceeded.
    Call log:
      - waiting for locator('#news-sidebar .sidebar-teaser form').locator('.form-wrapper').locator('.select-wrapper').nth(5).locator('a').filter({ hasText: /^Deutschland$/ })
      -   locator resolved to <a class="ng-binding" data-ng-click="wamData.areas.…>Deutschland</a>
      - attempting click action
      -   waiting for element to be visible, enabled and stable
      -     element is not visible - waiting...


      58 |     for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      59 |       await div.locator('span.icon-angle-down').click(); // show dropdown options
    > 60 |       await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
         |                                                                                   ^
      61 |       i++;
      62 |     }
      63 |     await form.locator('button:has-text("anzeigen")').click();

        at /usr/src/app/tests/00.spec.ts:60:83

Why are the tests sometimes successful and sometimes not? Is there a way to make them always successful? (Maybe improve the playwright settings somehow?)

Files:

Dockerfile

FROM mcr.microsoft.com/playwright:v1.41.1-jammy

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

ENTRYPOINT ["npx", "playwright", "test"]

package.json

{
  "name": "playwright-tests",
  "version": "1.0.0",
  "description": "Testing web access using Playwright",
  "scripts": {
    "test": "jest"
  },
  "devDependencies": {
    "@playwright/test": "^1.16.1",
    "jest": "^27.3.1"
  }
}

tests/00.spec.ts

import { test, expect } from '@playwright/test';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

test.beforeEach(async ({ page }) => {
  await page.goto('http://www.fussball.de');

  // accept cookies
  var ucr = await page.locator('#usercentrics-root');
  await ucr.locator('button:has-text("Akzeptiere alle")').click();
});
  
var leagues = {
  'https://www.fussball.de/spieltagsuebersicht/a-junioren-bundesliga-west-deutschland-a-junioren-bundesliga-a-junioren-saison2324-deutschland/-/staffel/02LNP13DU800000FVS5489B4VUAB0UC4-G#!/': [
    'Deutschland',
    '23/24',
    'Meisterschaften',
    'A-Junioren',
    'Bundesliga',
    'Deutschland',
    'A-Junioren-Bundesliga West'
  ],
  'https://www.fussball.de/spieltagsuebersicht/bfv-landesliga-odenwald-baden-landesliga-herren-saison2324-baden/-/staffel/02M5LMHKRK00000FVS5489B4VSAUO6GA-G#!/': [
    'Baden',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Baden',
    'bfv-Landesliga Odenwald'
  ],
  'https://www.fussball.de/spieltagsuebersicht/landesliga-mitte-bayern-landesliga-herren-saison2324-bayern/-/staffel/02LS70U7C4000007VS5489B4VUAB0UC4-G#!/': [
    'Bayern',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Bayern',
    'Landesliga Mitte'
  ],
  'https://www.fussball.de/spieltagsuebersicht/herren-landesliga-st1-berlin-landesliga-herren-saison2324-berlin/-/staffel/02M7480S2O000006VS5489B4VSAUO6GA-G#!/': [
    'Berlin',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Berlin',
    'Herren Landesliga St.1'
  ],
};

var j = 1;
for (const url of Object.keys(leagues)) {
  test('league form ' + j, async ({ page }) => {
    var form = await page.locator('#news-sidebar .sidebar-teaser form');
    var form_wrapper = await form.locator('.form-wrapper');
    var i = 0;
    for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      await div.locator('span.icon-angle-down').click(); // show dropdown options
      await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
      i++;
    }
    await form.locator('button:has-text("anzeigen")').click();
    await expect(page).toHaveURL(url);
  });

  j++;
}

tests/01.spec.ts

import { test, expect } from '@playwright/test';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

test.beforeEach(async ({ page }) => {
  await page.goto('http://www.fussball.de');

  // accept cookies
  var ucr = await page.locator('#usercentrics-root');
  await ucr.locator('button:has-text("Akzeptiere alle")').click();
});
  
var leagues = {
  'https://www.fussball.de/spieltagsuebersicht/a-junioren-bundesliga-west-deutschland-a-junioren-bundesliga-a-junioren-saison2324-deutschland/-/staffel/02LNP13DU800000FVS5489B4VUAB0UC4-G#!/': [
    'Deutschland',
    '23/24',
    'Meisterschaften',
    'A-Junioren',
    'Bundesliga',
    'Deutschland',
    'A-Junioren-Bundesliga West'
  ],
};

var j = 1;
for (const url of Object.keys(leagues)) {
  test('league form ' + j, async ({ page }) => {
    var form = await page.locator('#news-sidebar .sidebar-teaser form');
    var form_wrapper = await form.locator('.form-wrapper');
    var i = 0;
    for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      await div.locator('span.icon-angle-down').click(); // show dropdown options
      await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
      i++;
    }
    await form.locator('button:has-text("anzeigen")').click();
    await expect(page).toHaveURL(url);
  });

  j++;
}

tests/02.spec.ts

import { test, expect } from '@playwright/test';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

test.beforeEach(async ({ page }) => {
  await page.goto('http://www.fussball.de');

  // accept cookies
  var ucr = await page.locator('#usercentrics-root');
  await ucr.locator('button:has-text("Akzeptiere alle")').click();
});
  
var leagues = {
  'https://www.fussball.de/spieltagsuebersicht/bfv-landesliga-odenwald-baden-landesliga-herren-saison2324-baden/-/staffel/02M5LMHKRK00000FVS5489B4VSAUO6GA-G#!/': [
    'Baden',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Baden',
    'bfv-Landesliga Odenwald'
  ],
};

var j = 1;
for (const url of Object.keys(leagues)) {
  test('league form ' + j, async ({ page }) => {
    var form = await page.locator('#news-sidebar .sidebar-teaser form');
    var form_wrapper = await form.locator('.form-wrapper');
    var i = 0;
    for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      await div.locator('span.icon-angle-down').click(); // show dropdown options
      await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
      i++;
    }
    await form.locator('button:has-text("anzeigen")').click();
    await expect(page).toHaveURL(url);
  });

  j++;
}

tests/03.spec.ts

import { test, expect } from '@playwright/test';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

test.beforeEach(async ({ page }) => {
  await page.goto('http://www.fussball.de');

  // accept cookies
  var ucr = await page.locator('#usercentrics-root');
  await ucr.locator('button:has-text("Akzeptiere alle")').click();
});
  
var leagues = {
  'https://www.fussball.de/spieltagsuebersicht/landesliga-mitte-bayern-landesliga-herren-saison2324-bayern/-/staffel/02LS70U7C4000007VS5489B4VUAB0UC4-G#!/': [
    'Bayern',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Bayern',
    'Landesliga Mitte'
  ],
};

var j = 1;
for (const url of Object.keys(leagues)) {
  test('league form ' + j, async ({ page }) => {
    var form = await page.locator('#news-sidebar .sidebar-teaser form');
    var form_wrapper = await form.locator('.form-wrapper');
    var i = 0;
    for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      await div.locator('span.icon-angle-down').click(); // show dropdown options
      await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
      i++;
    }
    await form.locator('button:has-text("anzeigen")').click();
    await expect(page).toHaveURL(url);
  });

  j++;
}

tests/04.spec.ts

import { test, expect } from '@playwright/test';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

test.beforeEach(async ({ page }) => {
  await page.goto('http://www.fussball.de');

  // accept cookies
  var ucr = await page.locator('#usercentrics-root');
  await ucr.locator('button:has-text("Akzeptiere alle")').click();
});
  
var leagues = {
  'https://www.fussball.de/spieltagsuebersicht/herren-landesliga-st1-berlin-landesliga-herren-saison2324-berlin/-/staffel/02M7480S2O000006VS5489B4VSAUO6GA-G#!/': [
    'Berlin',
    '23/24',
    'Meisterschaften',
    'Herren',
    'Landesliga',
    'Berlin',
    'Herren Landesliga St.1'
  ],
};

var j = 1;
for (const url of Object.keys(leagues)) {
  test('league form ' + j, async ({ page }) => {
    var form = await page.locator('#news-sidebar .sidebar-teaser form');
    var form_wrapper = await form.locator('.form-wrapper');
    var i = 0;
    for (const div of await form_wrapper.locator('.select-wrapper').all()) {
      await div.locator('span.icon-angle-down').click(); // show dropdown options
      await div.locator('a', {hasText: new RegExp('^' + leagues[url][i] + '$') }).click(); // select option
      i++;
    }
    await form.locator('button:has-text("anzeigen")').click();
    await expect(page).toHaveURL(url);
  });

  j++;
}

Answer 1

The simple answer is that it's poorly written :) You have 4 equal tests that check the same thing with different test data, bad locators and no validation based on expect PW API.

I'd suggest to

1. Remove cookie banner interaction before every test. You can save cookies when it closed and use test.use with storage state
2. Rewrite locators like ('#news-sidebar .sidebar-teaser form').locator('.form-wrapper').locator('.select-wrapper').nth(5).locator('a') to something more clear.
3. Write better validation for your UI to prove the page is loaded. It's like API response catching. Or some UI element waiting.

If you don't like doing that, you have two options (bad pieces of advice here)

1. Don't run your test in parallel. npx playwright test --workers=1
2. Use slowMo in test use options in config.

You may try to use npx playwright codegen to generate the steps you may try to follow - that would give you a better understanding of locators usage in Playwright.

Good luck