ci: stop unrelated apt repos from gating the Chromium install - #3635
Conversation
`playwright install --with-deps` shells out to `apt-get update`, which consults every repository configured on the runner image and fails the whole install if any single one is unreachable. The ubuntu-24.04 image ships third-party repos that have nothing to do with Chromium, and packages.microsoft.com has been returning 403 Forbidden: E: Failed to fetch https://packages.microsoft.com/repos/azure-cli/dists/noble/InRelease 403 Forbidden E: Failed to fetch https://packages.microsoft.com/ubuntu/24.04/prod/dists/noble/InRelease 403 Forbidden Failed to install browsers The existing lock-wait/retry/timeout scaffolding cannot help: a 403 is a persistent server-side state, not a blip, so both attempts failed the same way 21 seconds apart. Park every apt source that is not an Ubuntu archive before installing, so only the repository that actually serves Chromium's system libraries can gate the job. Fail-safe: if no Ubuntu archive source is detected the sources are left untouched and a warning is emitted. Applied to both browser-installing jobs (rsc browser e2e, binary e2e).
|
Warning Review limit reached
Next review available in: 9 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Comment |
Why this is worth fixing
This is not a flake. The merge queue has been ejecting healthy PRs all day on
infrastructure noise, and every ejection teaches reviewers to re-queue without
reading the log. That habit is the actual risk: it is how a genuine failure
eventually gets waved through. Removing the noise source is what keeps the
signal trustworthy.
The failure
merge_group run
31574851781, jobtests (binary e2e), ejected #3626:Root cause
playwright install --with-deps chromiumshells out toapt-get update, whichconsults every repository configured on the runner image and fails the whole
command if any single one is unreachable. The
ubuntu-24.04image shipsthird-party repos —
packages.microsoft.com/repos/azure-cli,packages.microsoft.com/ubuntu/24.04/prod,dl.google.com, ... — that havenothing whatsoever to do with Chromium. When Microsoft's CDN returns 403, apt
exits non-zero, so the browser install fails, so the job fails.
Why the existing hardening did not help
The step already has a
wait_for_apt_locksloop, a 2-attempt retry withretry_backoff_seconds, and atimeoutwrapper. None of it can help here: a403 is a persistent server-side state, not a transient blip. Both attempts
failed identically, 21 seconds apart. Retrying a 403 just produces a second 403.
The fix
Option (a), generalised from "unbreak Microsoft" to "close the whole failure
class": before installing, park every apt source that is not an Ubuntu
archive. Chromium's system libraries come only from the Ubuntu archive, so
that is the only repo allowed to gate the job. A transient failure of any
unrelated third-party repo now cannot fail this step — today's Microsoft 403,
and equally a future
dl.google.comordownload.docker.comoutage.Fail-safe: if no Ubuntu archive source is detected, sources are left untouched
and a warning is emitted, so a future image layout change degrades to today's
behaviour rather than deleting the archive out from under apt.
Why not option (b), dropping
--with-depsThat would mean betting that the runner image happens to carry all ~21 Chromium
libraries, forever. I could not verify that claim without asserting it on a
runner, and even a verified snapshot would be verified only against
ubuntu24/20260720.247— a later image that drops a library would resurface asa cryptic "browser failed to launch" instead of a legible apt error. Keeping
--with-depskeeps the dependency set correct by construction; scoping apt'ssources removes the noise without giving that up.
Scaffolding kept
The lock-wait / retry / timeout scaffolding is retained and is still load
bearing — it now covers what it was actually good for: dpkg lock contention and
genuine transient failures of the Ubuntu archive itself. Nothing was removed.
Both call sites
Applied to both browser-installing jobs, since fixing one leaves the other
flaky:
tests (rsc browser e2e)—.github/workflows/cicd.ymltests (binary e2e)—.github/workflows/cicd.ymlVerification
bash -esemantics against areproduction of the real runner source set (
ubuntu.sources+azure-cli.list+microsoft-prod.list+google-chrome.list+docker.list): the four third-party sources are parked,ubuntu.sourcesiskept, step exits 0.
sources.list.d(glob does not expand literally), and re-run idempotency.Not weakened
No
continue-on-error, no skipped tests, no relaxed assertions. A real browserfailure or test failure still fails the build.