Skip to content

ci(webviews): replay base reds on the merge parent; room for a slow file's retry - #27194

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/web-base-check-merge-parent
Oct 29, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/web-base-check-merge-parent

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Webview test reds on #27045 (run 40781148854) came from CI, not from the change, which only touched HarnessesScreen.gallery.tsx, harnesses.css and harnesses-hover.test.ts. There were three causes, and this PR fixes all three.

1. react-apps-base-check replayed on the wrong base. It checked out github.event.pull_request.base.sha. That value lags the merge commit's first parent once feat-cmux-next moves. #27045's head shards ran on merge commit 09e00bee, whose parent is f0dc65f8. Its base job replayed on e275db3f, 12 commits older. That older commit doesn't have #27019's 999px workflow-card bars, so agent-pane-no-capsules passed there and its red was charged to #27045 (shard 4, job 123220786884). The base job now checks out the merge commit at depth 2 and detaches at HEAD^1. That is the same base the head shards' prScope diff uses, and the same fix #23534 needed in cmux-next-guards.yml.

2. The serial-retry budget was too short for the slowest file. Head shard 1 (job 123220840265) first ran agent-pane-attach-huge-files alone and got typing p95 32/46/39 ms against limits of 30/36/36. The runner's 120 s retry budget had already spent 60 s retrying code-block-highlight-height, so the huge-files retry was cut off at 60 s. The file needs about 130 s, so the first run's numbers were kept. The base replay ran the same file alone three times (two jobs on 32 vCPU, one on 16 vCPU) and it passed every time. Head shards now get CMUX_WEB_TEST_RETRY_BUDGET=300. The slowest shard (about 32 min on 16 vCPU) still fits in its 40 min.

3. composer-location-strip runs close to the default 120 s per-file limit. Alone, or on 32 vCPU, it takes 67-98 s. On a 16 vCPU shard next to 15 other files it timed out at 120 s, and timeouts are never retried. It now declares // cmux-web-test-timeout: 240. That is a per-file limit, not an excuse: a real timeout still fails.

A note on sharding: adding harnesses-hover.test.ts shifted the index-modulo shard split by one. That moved huge-files, composer-location-strip and code-block-highlight-height from shard 2 to shard 1, and agent-pane-no-capsules from shard 1 to shard 4. The base push run (40783304241) ran shard 2 on 32 vCPU, while #27045's head shards ran on 16 vCPU (the overflow fallback). The shard move only exposed fixes 2 and 3. It isn't the cause on its own, because the huge-files file already runs alone (// cmux-web-test-alone).

The attachment-size-cap work may also edit agent-pane-attach-huge-files.test.ts. This PR does not touch that file.

Tests:

  • tests/test_webviews_tests_new_reds.py: 2 new tests. The base job must not use pull_request.base.sha and must detach at HEAD^1 before installing. Head shards must carry the 300 s retry budget.
  • Run on a Linux host, all passing: test_webviews_tests_new_reds.py (22), test_webviews_tests_per_file.py (22), test_ci_blacksmith_linux_failover.py, test_check_paths.py, test_ci_change_areas.py, test_ci_linux_compile_runners.py, test_ci_workflow_run_sources.py, test_cmux_next_checks_workflow.py, test_cmux_next_web_base_push.py, test_supplychain_security.py, test_web_validation.py, test_ci_self_hosted_guard.sh. test_pr_scoped_webviews_checks.py and test_cmux_next_fast_gates.py error the same way on base, because that host has no environment for them.

Risk: CI only. On a pull request, the base replay now compares against the merge parent instead of the possibly stale base sha. That can turn a former "red on this change" into "red on the base too" only when the merge parent really has the red. The cost is a slower worst case for head shards (up to 3 more minutes of retries).

🤖 Generated with Claude Code

Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes webview test false reds by replaying base tests on the merge parent instead of a stale base SHA, and gives head shards room to retry slow files.

  • The base job now checks out the merge commit at depth 2 and detaches at HEAD^1, the same commit the head shards' diff uses; pull_request.base.sha lagged the first parent once the base moved, so the merge parent's own red was charged to the change.
  • Head shards get CMUX_WEB_TEST_RETRY_BUDGET=300. The old 120 s default cut off agent-pane-attach-huge-files's retry after a shorter retry used half the budget.
  • composer-location-strip.test.ts declares a 240 s per-file timeout; it hit the 120 s limit on a busy 16 vCPU shard, and timeouts are never retried.

Written for commit 111e888. Summary will update on new commits.

View guided diff Turn on auto-fix

…etry

react-apps-base-check checked out pull_request.base.sha, which lags the merge
commit's first parent once the base moves: #27045's replay ran 12 commits
behind the commit its head shards merged onto, where agent-pane-no-capsules
still passed, so #27019's capsule red was charged to the change. The job now
checks out the merge commit (depth 2) and detaches at HEAD^1, the same base the
head shards' prScope diff uses.

Head shards get a 300 s retry budget: at the runner's 120 s default, shard 1
spent 60 s retrying code-block-highlight-height and cut attach-huge-files'
alone retry (about 130 s) short, so its first run's typing reads stood.

composer-location-strip declares 240 s: it takes 67-98 s alone or on 32 vCPU
and timed out at 120 s beside 15 others on a 16 vCPU shard; timeouts are not
retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 111e888: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27194 -f ref=lane/web-base-check-merge-parent -f platform=macos -f backend_mode=local

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-99427x created for this PR by the cmux coordinator.

@teamleaderleo
teamleaderleo merged commit 9c6b4a2 into feat-cmux-next Oct 29, 2026
170 of 175 checks passed
@teamleaderleo
teamleaderleo deleted the lane/web-base-check-merge-parent branch October 29, 2026 03:20
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 29, 2026
… tested on a merge commit (#27202)

run-webviews-tests.sh never retried a TIMEOUT, and failed files' retries shared one budget. On
#27130's head shard 4, composer-footer-plain and new-tab-empty-one-line timed out at 120 s beside
15 other files and were never retried; alone, on the base replay, they passed in 61 s and 22 s.

Each timed-out file now gets one retry, alone, at its full limit, outside the failed files'
CMUX_WEB_TEST_RETRY_BUDGET and retry count, so one slow file cannot eat the others' retries.
Timeout retries have their own budget (CMUX_WEB_TEST_TIMEOUT_RETRY_BUDGET, default 240 s), and
one starts only when its whole limit fits, so the shard's worst case stays bounded. A retry that
times out again stays a TIMEOUT: never a pass and never a named failure a base could share.
CMUX_WEB_TEST_RETRIES=0 (the base replay) turns these retries off too.

The base replay's merge-parent checkout (#27194) gets a behavioural test. A real merge commit
whose first parent has a red file that the older pull_request.base.sha passed runs the base job's
own step scripts from ci-web.yml. At HEAD^1 the head red is a base red; replayed on the old base
it is charged to the change (#26980's message-preview-browser red held #27130, #27000, #26988
and #27008 that way).

Red/green: tests/test_webviews_tests_per_file.py's 3 new timeout tests fail on the old runner and
pass now. The new Workflow test fails when the step detaches at HEAD^2 instead of HEAD^1.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-7a592ef92c66f080e3cc

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 111e888c Deployed Oct 29, 2026 by teamleaderleo via cmux-tui tree preflight #74661
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant