[teamleaderleo]

Leo's ask, relayed: "Webview tests keep breaking base as fast as they get fixed. Leo wants them caught at PR time, as a system fix rather than more red-fix PRs."

Of the 22 PRs that brought a webview red to feat-cmux-next since 2026-10-28, 14 merged with react-apps-base-check finished and red, and 8 merged while it was still pending. Per-diff routing never skipped the suite. A blocking verdict can't be built on what the gates read today, for three reasons:

1. Flakes count as new reds. The base replay runs each red file once, on the merge parent. A test that fails now and then on the base can pass that one run by chance, and the red is then charged to the change. agent-pane-suggested-tasks was charged on 9 of 40 sampled red PR runs, and newtab-project-chip-fit the same way.
2. accent-glass-invariants timed out. It scans 2,542 stages in about 9 minutes alone, against a 900 s limit. On a busy shard it timed out with no room left before the retry deadline, so it was charged as new (runs 41079754472, 41076022470, 41048809560).
3. The red-checks comment called the wrapper checks "base red." That held even when a job they gather was the change's own red, and a red react-apps-base-check read "no finished base run to compare."

This PR fixes all three. It changes no workflow permissions and posts no new statuses. A follow-up gate (merge_guard / next-autoland) reads the existing web / react-apps-base-check (N) conclusion and log.

Changes

• scripts/ci/webviews_test_history.py (new):
  • record adds one push run's per-test webview verdicts (from the shards' results) to a history of the last 120 runs, with the run's inputs key.
  • The inputs key is the git ids of every path web_subareas.py routes the webview suite on.
  • flaky names a test only when two base pushes with the same inputs gave it different verdicts. Most pushes change no webview input, so that evidence builds up fast. A break and its revert (green, red, green) change the inputs each time, so they are never flaky, and a re-run of a push counts as more evidence.
  • A file that timed out says nothing about its tests on that run.
  • web_subareas.py now routes the suite on for this script too.
• ci-web.yml:
  • On pushes, each shard uploads its results.
  • A new push-only upkeep job, webviews-test-history, saves the history to the Actions cache under webviews-test-history-<branch>-<run>. Every step in it is continue-on-error, like webviews-test-durations.
  • react-apps-base-check restores the base branch's history with read-only actions/cache/restore and passes the flaky names to the comparison. PR runs never save it, so a PR can't write its base's history.
  • The base check now runs the head's copy of guards_new_reds.py and of the history script, which the head shard publishes beside its runner, as it already does for run-webviews-tests.sh.
• guards_new_reds.py --flaky FILE: a listed red that the base replay passed becomes ::warning title=Webview test flaky on the base:: and the step stays green. Every other new red is still ::error ... red on this change:: and fails the step.
• accent-glass-invariants is split into four parts (accent-glass-invariants-1..4.test.ts, 480 s each). Each part scans every fourth stage of the sorted list. The body moved unchanged to test/support/accentGlassInvariants.ts. The stale-baseline warning covers only the part's own stages.
• pr_red_checks_comment.py:
  • web / react-apps-check and web / Web status are never called base reds. Each follows only the jobs it gathers: react-apps-check follows its shards, the base replays and react-doctor; Web status follows the other web / jobs.
  • An aggregate blocks when a job it gathers blocks, when one of them was cancelled, or when none of them is red. It is excused only when every red job it gathers is a base red. A red it doesn't gather, such as the size cap, never excuses it.
  • A red react-apps-base-check (N) names up to 3 of its new red tests, read from its "Webview test red on this change" annotations, and blocks.

Repo rule note: CLAUDE.md asks not to add CI jobs or steps. The new job and steps are upkeep, never a gate, and nothing in them can fail a run. Leo's standing orders make CI changes the lanes' call after independent review.

Until this lands, base-check PR runs use the base's scripts, so the flake handling takes effect on PRs opened after it merges. The history fills from the first feat-cmux-next pushes after that, and a flake needs two runs of the test on the same inputs.

Observed / Not run

• Codex review (gpt-6-astra) BLOCKed twice, and both are fixed:
  • stage names with spaces in the accent-glass baseline;
  • break-and-revert treated as a flake, and aggregates excused by unrelated base reds.
• Ran here: tests/test_webviews_test_history.py (16 OK, new), tests/test_guards_new_reds.py (OK, with 3 new --flaky cases), tests/test_webviews_tests_new_reds.py (OK; the workflow-step tests now publish the two scripts with the runner), tests/test_pr_red_checks_comment.py (27 PASS, 7 new), tests/test_cmux_next_web_base_push.py, tests/test_webviews_tests_per_file.py, test_runner_label_policy, test_ci_runs_on_values, test_ci_runs_on_fallback, test_runs_on_labels, test_ci_blacksmith_linux_failover, test_ci_pr_runner_pool, test_ci_change_areas and test_test_reds (all OK).
• tests/test_ci_fork_runner_routing.py fails with the same single failure on the base (cmux-next.yml:1304 CI_SESSION_LANE_RUNNER), so it isn't this change's.
• The last commit wires tests/test_webviews_test_history.py into cmux-next.yml's guard list (the every-guard-is-wired check) and gives the history upload step an id the gate reads as a warning (the continue-on-error id check).
• CI at 31a587e: every check green, including the four react-apps-check shards (they run the four accent-glass parts) and all four react-apps-base-check replays. Codex review (gpt-6-astra): SHIP at d5ca5b4, then SHIP on the delta to 31a587e.

risk: normal (CI scripts and tests; no product code).

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 31a587e: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28511 -f ref=dep-fleet/webview-flake-verdict -f platform=macos -f backend_mode=local

[teamleaderleo]

Owned by an agent session · session

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[blacksmith-sh]

Found 13 test failures on Blacksmith runners:

Failures

Test	View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed:	View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed:	View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/ test_download_prints_the_active_profile_only	View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/ Security has no stub	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/ CoreFoundation, missing 1: ['_CFRetain']	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/ CoreFoundation, missing 1: ['_log2']	View Logs
test_big_red (main.FleetRuleLints.test_big_red)/ test_big_red (main.FleetRuleLints.test_big_red)	View Logs
test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_ every_upload_is_short_lived_and_small_or_allowlisted)/ test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_ every_upload_is_short_lived_and_small_or_allowlisted)	View Logs
test_fork_pull_requests_into_manaflow_ai_never_reach_an_owned_runner_label (main.Fo rkRunnerRoutingTests.test_fork_pull_requests_into_manaflow_ai_never_reach_an_owned_runn er_label)/ test_fork_pull_requests_into_manaflow_ai_never_reach_an_owned_runner_label (main.Fo rkRunnerRoutingTests.test_fork_pull_requests_into_manaflow_ai_never_reach_an_owned_runn er_label)	View Logs
test_no_web_check_waits_on_an_earlier_red (main.WorkflowShape.test_no_web_check_wai ts_on_an_earlier_red) (gated="Upload this push's webview results")/ test_no_web_check_waits_on_an_earlier_red (main.WorkflowShape.test_no_web_check_wai ts_on_an_earlier_red) (gated="Upload this push's webview results")	View Logs

...and 3 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4113397306901 (dd50ec33ed)
• cmux NEXT nightly 1.0.0-nightly.4113454748501 (a94df426d7)
• cmux NEXT nightly 1.0.0-nightly.4113616106401 (901c76981f)