Skip to content

ci: stop unrelated apt repos from gating the Chromium install - #3635

Merged
kojiwakayama merged 1 commit into
mainfrom
fix/ci-browser-install-apt-isolation
Aug 12, 2026
Merged

kojiwakayama merged 1 commit into
mainfrom
fix/ci-browser-install-apt-isolation

Conversation

@kojiwakayama

Copy link
Copy Markdown
Contributor

Why this is worth fixing

This is not a flake. The merge queue has been ejecting healthy PRs all day on
infrastructure noise, and every ejection teaches reviewers to re-queue without
reading the log. That habit is the actual risk: it is how a genuine failure
eventually gets waved through. Removing the noise source is what keeps the
signal trustworthy.

The failure

merge_group run 31574851781, job tests (binary e2e), ejected #3626:

E: Failed to fetch https://packages.microsoft.com/repos/azure-cli/dists/noble/InRelease  403  Forbidden
E: Failed to fetch https://packages.microsoft.com/ubuntu/24.04/prod/dists/noble/InRelease  403  Forbidden
Failed to install browsers
Error: Installation process exited with code: 100

Root cause

playwright install --with-deps chromium shells out to apt-get update, which
consults every repository configured on the runner image and fails the whole
command if any single one is unreachable. The ubuntu-24.04 image ships
third-party repos — packages.microsoft.com/repos/azure-cli,
packages.microsoft.com/ubuntu/24.04/prod, dl.google.com, ... — that have
nothing whatsoever to do with Chromium. When Microsoft's CDN returns 403, apt
exits non-zero, so the browser install fails, so the job fails.

Why the existing hardening did not help

The step already has a wait_for_apt_locks loop, a 2-attempt retry with
retry_backoff_seconds, and a timeout wrapper. None of it can help here: a
403 is a persistent server-side state, not a transient blip. Both attempts
failed identically, 21 seconds apart. Retrying a 403 just produces a second 403.

The fix

Option (a), generalised from "unbreak Microsoft" to "close the whole failure
class": before installing, park every apt source that is not an Ubuntu
archive. Chromium's system libraries come only from the Ubuntu archive, so
that is the only repo allowed to gate the job. A transient failure of any
unrelated third-party repo now cannot fail this step — today's Microsoft 403,
and equally a future dl.google.com or download.docker.com outage.

Fail-safe: if no Ubuntu archive source is detected, sources are left untouched
and a warning is emitted, so a future image layout change degrades to today's
behaviour rather than deleting the archive out from under apt.

Why not option (b), dropping --with-deps

That would mean betting that the runner image happens to carry all ~21 Chromium
libraries, forever. I could not verify that claim without asserting it on a
runner, and even a verified snapshot would be verified only against
ubuntu24/20260720.247 — a later image that drops a library would resurface as
a cryptic "browser failed to launch" instead of a legible apt error. Keeping
--with-deps keeps the dependency set correct by construction; scoping apt's
sources removes the noise without giving that up.

Scaffolding kept

The lock-wait / retry / timeout scaffolding is retained and is still load
bearing — it now covers what it was actually good for: dpkg lock contention and
genuine transient failures of the Ubuntu archive itself. Nothing was removed.

Both call sites

Applied to both browser-installing jobs, since fixing one leaves the other
flaky:

  • tests (rsc browser e2e).github/workflows/cicd.yml
  • tests (binary e2e).github/workflows/cicd.yml

Verification

  • Step logic exercised locally under GitHub's bash -e semantics against a
    reproduction of the real runner source set (ubuntu.sources +
    azure-cli.list + microsoft-prod.list + google-chrome.list +
    docker.list): the four third-party sources are parked, ubuntu.sources is
    kept, step exits 0.
  • Edge cases confirmed: no-Ubuntu-source (warns, touches nothing), empty
    sources.list.d (glob does not expand literally), and re-run idempotency.
  • Confirmed no later step in either job depends on a parked repository.

Not weakened

No continue-on-error, no skipped tests, no relaxed assertions. A real browser
failure or test failure still fails the build.

`playwright install --with-deps` shells out to `apt-get update`, which
consults every repository configured on the runner image and fails the
whole install if any single one is unreachable. The ubuntu-24.04 image
ships third-party repos that have nothing to do with Chromium, and
packages.microsoft.com has been returning 403 Forbidden:

  E: Failed to fetch https://packages.microsoft.com/repos/azure-cli/dists/noble/InRelease  403  Forbidden
  E: Failed to fetch https://packages.microsoft.com/ubuntu/24.04/prod/dists/noble/InRelease  403  Forbidden
  Failed to install browsers

The existing lock-wait/retry/timeout scaffolding cannot help: a 403 is a
persistent server-side state, not a blip, so both attempts failed the
same way 21 seconds apart.

Park every apt source that is not an Ubuntu archive before installing, so
only the repository that actually serves Chromium's system libraries can
gate the job. Fail-safe: if no Ubuntu archive source is detected the
sources are left untouched and a warning is emitted.

Applied to both browser-installing jobs (rsc browser e2e, binary e2e).
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@kojiwakayama, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 11538b0a-b8ab-4e9d-96e2-489e8f89b8de

📥 Commits

Reviewing files that changed from the base of the PR and between 1ca03fa and dfacac4.

📒 Files selected for processing (1)
  • .github/workflows/cicd.yml

Comment @coderabbitai help to get the list of available commands.

@kojiwakayama
kojiwakayama added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit d9814a5 Aug 12, 2026
33 checks passed
@kojiwakayama
kojiwakayama deleted the fix/ci-browser-install-apt-isolation branch August 12, 2026 09:55
@kojiwakayama kojiwakayama mentioned this pull request Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant