Tests failing with Firefox 135.0

[kptdobe]

Our Firefox started failing for a couple of days. I am not sure if this is a runner issue or a playwright issue.

This is happening with Firefox 135.0 (playwright build v1475).

I have created a repo with a basic setup to illustrate the bug.

For Chrome and locally, all tests are green. Note that it only happens in the GitHub action.

As soon as there is more than one file to run test on, it fails on Firefox.

Scenario 1: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816836084

1. 2 most basic test files (one .js and one .html)
2. Success: testing files individually works fine
3. Unexpected failure: testing the 2 files together fails

Scenario 2: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816896380

1. I have incremented the browserStartTimeout as recommended - set to 2mins...
2. Success: testing files individually works fine
3. Unexpected failure: testing the 2 files together fails

Scenario 3: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13816945697

1. I have added 2 more basic test files (2 .js and 2 .html)
2. Unexpected success: the test that failed before has now passed...
3. Unexpected failures: the 2 combos (2x testing JS files or 2x testing the HTML files), fails. This was to eliminate the case that maybe it is the JS+HTML test files.

Scenario 4: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13817246076

1. I have incremented the timeout to 4mins
2. Success: testing files individually works fine
3. Unexpected failure: all "combo" tests failed

The fact that the test on 3.2 passed reveals that "sometimes" it works. Increasing the timeout does not solve the problem. There is something else...

[kptdobe]

Scenario 5:
https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13830687684 (4 mins timeout)
https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13830722384 (no timeout)

1. downgraded to previous versions: Firefox 134.0 via "@web/test-runner": "0.19.0" and "@playwright/test": "1.50.1"
2. in both cases, all test succeeded except the 2 HTML files

Still not sure what happens but clearly, the problem was already there before. Not sure why no one is seeing this...

[bashmish]

Can you please check if the issue remains with the configuration concurrency: 1?

[kptdobe]

Weird, I had tested it before, did not work. But now it seems to roll, all tests green: https://github.com/kptdobe/web-test-runner-firefox-issue/actions/runs/13901645165

Nice temporary workaround, thanks @bashmish.

The problem is fixed on the playwright canary version (i.e. without concurrency config)

[bashmish]

I didn't fully understand what solved it for you. Was it concurrency: 1 or new canary playwright or both?
A bit more clarity can help others who run into this problem too.

For the context: concurrency: 1 is often solving issues like this, because the issues come from the fact the modern browsers significantly slow down inactive tabs and "block" certain events in them, while WTR uses multiple tabs to parallelize the execution of individual test files.concurrency: 1 practically means "use maximum 1 tab", which then will be active.

Also, you can close this issue if solved :)

[kptdobe]

Sorry for lack of clarity:

• I have created a new test scenario and all tests are green using concurrency: 1.
• but in the test, I also have browserStartTimeout: 240000 in the config, thus it is probably the combo of the 2 configs.

Using the 2 configs is a decent workaround.

In microsoft/playwright#35183, the playwright team mentioned the problem is fixed with their latest daily built, it seems something has been changed at the Firefox level. This means that when [email protected] is out, the workaround above will not be necessary.

I am more in favor in keeping the issue opened until it is "really" fixed. I'll test [email protected] to confirm and then will close this issue. If someone finds this issue, a closed issue sends the wrong message because workaround still needed.