Switch to new Chrome headless mode

[colinrotherham]

Following #3542 just a quick PR for:

• Switch to new Chrome headless mode #3541

For more info on our headless mode deprecation issues see:

• [Bug]: Repetitive deprecation warnings for { headless: true } puppeteer/puppeteer#10070

[colinrotherham]

Timing stats using Chrome for Testing on Puppeteer v20

Note: Percy previously downloaded its own Chromium even though we used Puppeteer

Before

Using old headless mode { headless: true }

Accessibility tests: 1m 53s
Behaviour tests: 31s
Component tests: 39s
Percy screenshots: 46s

See GitHub Action run

After

Using new headless mode { headless: 'new' }

Accessibility tests: 2m 8s
Behaviour tests: 31s
Component tests: 36s
Percy screenshots: 37s

See GitHub Action run

[36degrees]

Do you know if the Puppeteer team are aware that tests running under the new headless modes are slower? Is it something they're working to improve?

Sharding the accessibility tests is a neat idea (TIL that's a thing!) but I feel like our GitHub Actions are starting to get a little complex. Wondering if we should either wait until the performance of the new headless mode improves, or just accept that our tests will be a little bit slower for a while?

[colinrotherham]

Do you know if the Puppeteer team are aware that tests running under the new headless modes are slower? Is it something they're working to improve?

From some of the comments they're aware of it (approx 30% slower), but appreciate it's now the same full browser for both headless and headful modes

Sharding the accessibility tests is a neat idea (TIL that's a thing!) but I feel like our GitHub Actions are starting to get a little complex. Wondering if we should either wait until the performance of the new headless mode improves, or just accept that our tests will be a little bit slower for a while?

Ha that's great. I'd always wanted to try --shard out and this seemed perfect for it. That said, I'm more than happy to drop the commit (especially as it changes required status checks again!) or even hold off "new headless" for a while

For context this PR was thinking our { headless: 1 } loophole to silence the deprecation warning would be closed:

• [Bug]: Repetitive deprecation warnings for { headless: true } puppeteer/puppeteer#10070

But it's still holding strong

[colinrotherham]

I've rebased with #3491 and removed the jest --shard commit

No rush to approve this one but good to know it works when we need it

[OrKoN]

The second timing block in #3543 (comment) is the "After" one (both say "before")?

[colinrotherham]

The second timing block in #3543 (comment) is the "After" one (both say "before")?

@OrKoN Ahh thanks, updated!

[colinrotherham]

@36degrees I've updated the timing stats after rebasing with Puppeteer v20:

• Bump puppeteer from 19.11.1 to 20.1.1 #3589

The switch to Chrome for Testing (not Chromium) has closed the gap 🙌

[colinrotherham]

Although since the ~600 accessibility tests merged we've drifted a lot from the 2m 30s sweet spot we once had

[romaricpascal]

The timings now look pretty much the same as on main, let's merge this 🥳 ⛵