Repository navigation
ci(webviews): replay base reds on the merge parent; room for a slow file's retry - #27194
Merged
Merged
Conversation
…etry react-apps-base-check checked out pull_request.base.sha, which lags the merge commit's first parent once the base moves: #27045's replay ran 12 commits behind the commit its head shards merged onto, where agent-pane-no-capsules still passed, so #27019's capsule red was charged to the change. The job now checks out the merge commit (depth 2) and detaches at HEAD^1, the same base the head shards' prScope diff uses. Head shards get a 300 s retry budget: at the runner's 120 s default, shard 1 spent 60 s retrying code-block-highlight-height and cut attach-huge-files' alone retry (about 130 s) short, so its first run's typing reads stood. composer-location-strip declares 240 s: it takes 67-98 s alone or on 32 vCPU and timed out at 120 s beside 15 others on a 16 vCPU shard; timeouts are not retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
|
Collaborator
Author
|
Bead |
This was referenced Oct 29, 2026
Contributor
|
Shipped in:
|
teamleaderleo
added a commit
that referenced
this pull request
Oct 29, 2026
… tested on a merge commit (#27202) run-webviews-tests.sh never retried a TIMEOUT, and failed files' retries shared one budget. On #27130's head shard 4, composer-footer-plain and new-tab-empty-one-line timed out at 120 s beside 15 other files and were never retried; alone, on the base replay, they passed in 61 s and 22 s. Each timed-out file now gets one retry, alone, at its full limit, outside the failed files' CMUX_WEB_TEST_RETRY_BUDGET and retry count, so one slow file cannot eat the others' retries. Timeout retries have their own budget (CMUX_WEB_TEST_TIMEOUT_RETRY_BUDGET, default 240 s), and one starts only when its whole limit fits, so the shard's worst case stays bounded. A retry that times out again stays a TIMEOUT: never a pass and never a named failure a base could share. CMUX_WEB_TEST_RETRIES=0 (the base replay) turns these retries off too. The base replay's merge-parent checkout (#27194) gets a behavioural test. A real merge commit whose first parent has a red file that the older pull_request.base.sha passed runs the base job's own step scripts from ci-web.yml. At HEAD^1 the head red is a base red; replayed on the old base it is charged to the change (#26980's message-preview-browser red held #27130, #27000, #26988 and #27008 that way). Red/green: tests/test_webviews_tests_per_file.py's 3 new timeout tests fail on the old runner and pass now. The new Workflow test fails when the step detaches at HEAD^2 instead of HEAD^1. Agent-Session: https://cmux.com/deeplink/agent?id=aid-7a592ef92c66f080e3cc Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Webview test reds on #27045 (run 40781148854) came from CI, not from the change, which only touched
HarnessesScreen.gallery.tsx,harnesses.cssandharnesses-hover.test.ts. There were three causes, and this PR fixes all three.1.
react-apps-base-checkreplayed on the wrong base. It checked outgithub.event.pull_request.base.sha. That value lags the merge commit's first parent oncefeat-cmux-nextmoves. #27045's head shards ran on merge commit09e00bee, whose parent isf0dc65f8. Its base job replayed one275db3f, 12 commits older. That older commit doesn't have #27019's 999px workflow-card bars, soagent-pane-no-capsulespassed there and its red was charged to #27045 (shard 4, job 123220786884). The base job now checks out the merge commit at depth 2 and detaches atHEAD^1. That is the same base the head shards' prScope diff uses, and the same fix #23534 needed incmux-next-guards.yml.2. The serial-retry budget was too short for the slowest file. Head shard 1 (job 123220840265) first ran
agent-pane-attach-huge-filesalone and got typing p95 32/46/39 ms against limits of 30/36/36. The runner's 120 s retry budget had already spent 60 s retryingcode-block-highlight-height, so the huge-files retry was cut off at 60 s. The file needs about 130 s, so the first run's numbers were kept. The base replay ran the same file alone three times (two jobs on 32 vCPU, one on 16 vCPU) and it passed every time. Head shards now getCMUX_WEB_TEST_RETRY_BUDGET=300. The slowest shard (about 32 min on 16 vCPU) still fits in its 40 min.3.
composer-location-stripruns close to the default 120 s per-file limit. Alone, or on 32 vCPU, it takes 67-98 s. On a 16 vCPU shard next to 15 other files it timed out at 120 s, and timeouts are never retried. It now declares// cmux-web-test-timeout: 240. That is a per-file limit, not an excuse: a real timeout still fails.A note on sharding: adding
harnesses-hover.test.tsshifted the index-modulo shard split by one. That moved huge-files, composer-location-strip and code-block-highlight-height from shard 2 to shard 1, and agent-pane-no-capsules from shard 1 to shard 4. The base push run (40783304241) ran shard 2 on 32 vCPU, while #27045's head shards ran on 16 vCPU (the overflow fallback). The shard move only exposed fixes 2 and 3. It isn't the cause on its own, because the huge-files file already runs alone (// cmux-web-test-alone).The attachment-size-cap work may also edit
agent-pane-attach-huge-files.test.ts. This PR does not touch that file.Tests:
tests/test_webviews_tests_new_reds.py: 2 new tests. The base job must not usepull_request.base.shaand must detach atHEAD^1before installing. Head shards must carry the 300 s retry budget.test_webviews_tests_new_reds.py(22),test_webviews_tests_per_file.py(22),test_ci_blacksmith_linux_failover.py,test_check_paths.py,test_ci_change_areas.py,test_ci_linux_compile_runners.py,test_ci_workflow_run_sources.py,test_cmux_next_checks_workflow.py,test_cmux_next_web_base_push.py,test_supplychain_security.py,test_web_validation.py,test_ci_self_hosted_guard.sh.test_pr_scoped_webviews_checks.pyandtest_cmux_next_fast_gates.pyerror the same way on base, because that host has no environment for them.Risk: CI only. On a pull request, the base replay now compares against the merge parent instead of the possibly stale base sha. That can turn a former "red on this change" into "red on the base too" only when the merge parent really has the red. The cost is a slower worst case for head shards (up to 3 more minutes of retries).
🤖 Generated with Claude Code
Agent session: agent session
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Fixes webview test false reds by replaying base tests on the merge parent instead of a stale base SHA, and gives head shards room to retry slow files.
HEAD^1, the same commit the head shards' diff uses;pull_request.base.shalagged the first parent once the base moved, so the merge parent's own red was charged to the change.CMUX_WEB_TEST_RETRY_BUDGET=300. The old 120 s default cut offagent-pane-attach-huge-files's retry after a shorter retry used half the budget.composer-location-strip.test.tsdeclares a 240 s per-file timeout; it hit the 120 s limit on a busy 16 vCPU shard, and timeouts are never retried.Written for commit 111e888. Summary will update on new commits.