[BUG] Missing await in test with `test.fail()` causes other tests to be skipped

[lukaw3d]

System info

• Playwright Version: 1.38.0-alpha-aug-11-2023
• Operating System: Ubuntu 20
• Browser: All
• Other info:

Source code

• I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

https://github.com/lukaw3d/playwright-skip-tests-bug

Main parts:

• https://github.com/lukaw3d/playwright-skip-tests-bug/blob/main/tests/test2.spec.ts#L3-L6

  test('test2 (expect fail)', async ({ page }) => {
    test.fail()
    /* await */ expect(Promise.resolve('a')).resolves.toBe('b')
  })

• run with --workers 1

Expected

$ playwright test --workers 1 --reporter list
  ✓  1 tests/test1.spec.ts:3:5 › test1 (145ms)
  ✘  2 tests/test2.spec.ts:3:5 › test2 (expect fail) (59ms)
  ✓  3 tests/test3.spec.ts:3:5 › test3 (60ms)
  ✓  4 tests/test4.spec.ts:3:5 › test4 (74ms)
  ✓  5 tests/test5.spec.ts:3:5 › test5 (65ms)

or
any kind of a warning or error that something went wrong

Actual

$ playwright test --workers 1 --reporter list
  ✓  1 tests/test1.spec.ts:3:5 › test1 (145ms)
  ✘  2 tests/test2.spec.ts:3:5 › test2 (expect fail) (59ms)
  -  3 tests/test3.spec.ts:3:5 › test3
  -  4 tests/test4.spec.ts:3:5 › test4
  -  5 tests/test5.spec.ts:3:5 › test5

and command exists as success!

[lukaw3d]

Another way to encounter this behaviour (probably the same underneath):

test('test2 (expect fail)', async ({ page }) => {
  test.fail()
  page.on('domcontentloaded', () => {
    expect('a').toBe('b')
  })
  await page.goto('https://example.com/', { waitUntil: 'domcontentloaded' })
})

[lukaw3d]

This might be related to #11666