Skip to content

[cx-vte03] ci(webviews): cap browser file timeouts and serial retry time - #25548

Merged
teamleaderleo merged 9 commits into
feat-cmux-nextfrom
nx-ci-browser-shard-timeouts
Oct 26, 2026
Merged

teamleaderleo merged 9 commits into
feat-cmux-nextfrom
nx-ci-browser-shard-timeouts

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

The browser shard can outlive its 40-minute job: agent-pane-attach-huge-files.test.ts declares a 3,000-second file allowance, other sweeps declare 600–900 seconds, and up to 60 failed files are retried serially. On #25304, run 40368616367 was cancelled without writing its head-results artifact. That shard reported 23 first-pass failures; its 900-second gallery sweep took about 11 minutes, and silent serial retries then ran for 17 minutes before cancellation. The 3,000-second stress-file allowance is a separate ceiling violation, not identified as the active file in this cancelled shard.

The runner now caps each file at 900 seconds, reports its name before launch and failure log as soon as it finishes, keeps TIMEOUT failures without another serial attempt, and limits the whole retry phase, including an in-flight attempt, to a 120-second shared budget. Every selected file still runs; failures left after the retry budget remain failures. Intentional standalone stress runs can explicitly raise CMUX_WEB_TEST_MAX_FILE_TIMEOUT. An incomplete retry retains the first named failure. Bun's default per-test timeout remains 30 seconds, enclosed by the hard file deadline even if a suite overrides it. The existing results artifact also carries the exact runner and result parser for base replay, so the head and base use the same deadline policy with their own tests and Bun version. PR path routing is unchanged.

claude-sr2 requested: “give every test a hard timeout so a hang turns into a named failure instead of a 37-min cancel.” This addresses the base blocker seen on #25304 and #25325. claimPaint is a separate false-positive: it finishes in 1.76s and counts two correctly disabled ordered-list boundary buttons. Existing #24469 owns that repair; its Codex owner has resumed it.

Regression command: python3 tests/test_webviews_tests_per_file.py PerFileRunner.test_a_declared_long_file_cannot_exceed_the_runner_hard_limit PerFileRunner.test_a_timed_out_file_reports_once_without_a_serial_retry PerFileRunner.test_serial_retries_have_a_shared_time_budget. The new tests fail at regression commit dd84051fdbf915 over base c80354516c120e04a3adba32258268611a5e1a73; all pass with the fix.

Additional red/green regressions cover an in-flight retry exceeding the remaining budget (7781af1dac4340), keeping its original named failure when that retry is incomplete (29826b3de600f4), and executing head/base workflow scripts with a stale base runner (af0430b3b59c7a). python3 tests/test_webviews_tests_per_file.py and python3 tests/test_webviews_tests_new_reds.py pass on the final head.

Changelog

none

Observed / Not run

  • 19 per-file runner tests: passed, 26.086s.
  • 20 new-red comparison wrapper/workflow tests: passed, 20.441s.
  • bash scripts/ci/tests/run-webviews-tests-retry.test.sh: passed.
  • bash -n scripts/ci/run-webviews-tests.sh, diff check and prepush gate: passed.
  • actionlint .github/workflows/ci-web.yml: passed.
  • No native builds. Hosted shard validation on base and two webview PRs is still pending; no completion claim from a skipped check.
  • Independent Codex xhigh review held the initial head for mixed head/base deadline policies. That finding is repaired and the independent review returned Review (Codex xhigh): SHIP at 1736adf93024567e0c596b9df56360c48f30a1ab after rerunning all 39 focused tests, retry shell, actionlint and prepush gate.

Agent session: agent session

Base c803545: focused per-file runner regressions fail because a 3000-second header bypasses the hard ceiling, TIMEOUT files rerun serially, and serial retries have no shared budget. No native builds.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
The 3000s attachment stress-file allowance exceeded a whole40min PR job. Clamp declared/default file limits at300s unless an intentional standalone run raises CMUX_WEB_TEST_MAX_FILE_TIMEOUT; report START and failure detail immediately. Keep timed-out files red without retrying and stop starting serial retries after a120s shared budget. Focused regressions failed at dd84051; all per-file runner tests now pass. Bash retry coverage passes; no native builds.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 0f27fcc: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25548 -f ref=nx-ci-browser-shard-timeouts -f platform=macos -f backend_mode=local

Yours to fix: next-docs.test.ts (this PR changes the web test runner (.github/workflows/ci-web.yml)). Also failing here, not on main: composer-paste-images.test.tsx, composer-paste-images.test.tsx, chat-minimap.test.tsx and 8 more. Seen on other PRs too (likely flaky): chat-environment-scroll-stability.test.tsx, chat-environment-scroll-stability.test.tsx, composer-paste-images.test.tsx and 18 more. This PR merged before its CI finished: fix forward on main.

CI failed on 1736adf930 (run 40384050518 attempt 1): 5 code, 1 unknown.

Job Verdict Why
web / Web tests (2/4) code a web test failed
web / react-apps-base-check (3) code a web test failed
web / react-apps-base-check (4) code a web test failed
web / react-apps-base-check (2) code a web test failed
web / react-apps-base-check (1) code a web test failed
web / react-apps-check unknown no known signature; failed step: Check webview test shards

Failures:

  • yours next-docs.test.ts cmux-next docs > the CLI and MCP reference is fresh against the action catalog: failed (job)
  • seen on other PRs chat-environment-scroll-stability.test.tsx long-chat scroll leaves the Left chip mounted and stable, while own cwd changes update it: failed (job)
  • seen on other PRs chat-environment-scroll-stability.test.tsx long-chat scroll leaves the Right chip mounted and stable, while own cwd changes update it: failed (job)
  • seen on other PRs composer-paste-images.test.tsx pasting two copied images makes two chips and puts nothing in the prompt: failed (job)
  • seen on other PRs composer-paste-images.test.tsx an image copied as HTML alone (no file on the clipboard) still comes in as a chip, its text as text: failed (job)
  • seen on other PRs composer-paste-images.test.tsx a pasted image goes to the agent as an image block; the prompt's text has no data: URI: failed (job)
  • seen on other PRs composer-paste-images.test.tsx an image copied as HTML into a plain text field comes in as a chip and the text still goes in: failed (job)
  • new in this PR composer-paste-images.test.tsx a prompt copied from a bubble with '![…](data:…)' text pastes as a chip and its text, nothing inline: failed (job)
  • new in this PR composer-paste-images.test.tsx the same text pasted into a plain text field leaves its text there without the image: failed (job)
  • seen on other PRs claimPaint.test.tsx a pooled Settings page claimed by the host reports its paint once its first screen is in: failed (job)
  • ... 23 more in the job logs
Matched log lines
web / Web tests (2/4): (fail) cmux-next docs > the CLI and MCP reference is fresh against the action catalog [3.02ms]
web / react-apps-base-check (3): (fail) long-chat scroll leaves the Left chip mounted and stable, while own cwd changes update it [21019.65ms]
web / react-apps-base-check (4): (fail) a pooled Settings page claimed by the host reports its paint once its first screen is in [326.16ms]
web / react-apps-base-check (2): (fail) agent pane page > rounds its controls as small-radius rects, not capsules [6.42ms]
web / react-apps-base-check (1): (fail) chromium > the gutter beside every part of the stack scrolls the transcript, and the part keeps the pointer [2971.75ms]

Not re-run automatically: web / Web tests (2/4), web / react-apps-base-check (3), web / react-apps-base-check (4), web / react-apps-base-check (2), web / react-apps-base-check (1) are not machine failures.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: web / Web status; base red: the same failure on the base c803545; does not block
red: web / Web tests (2/4); no finished base run on c803545 to compare; do not merge until green; see lane-rules
red: web / react-apps-base-check (1); no finished base run on c803545 to compare; do not merge until green; see lane-rules
red: web / react-apps-base-check (2); no finished base run on c803545 to compare; do not merge until green; see lane-rules
red: web / react-apps-base-check (3); no finished base run on c803545 to compare; do not merge until green; see lane-rules
red: web / react-apps-base-check (4); no finished base run on c803545 to compare; do not merge until green; see lane-rules
red: web / react-apps-check; base red: the same failure on the base c803545; does not block

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo teamleaderleo added the risk:low Risk tier (docs/ci/risk-tiers.md) label Oct 26, 2026
@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:low Risk tier (docs/ci/risk-tiers.md) labels Oct 26, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-vte03 created for this PR by the cmux coordinator.

At0f27fcc0 the shared budget only stops new retries: an already-started4-second retry outlives a1-second budget. The focused regression fails on elapsed runtime.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
Focused regression at7781af1d:4.137s retry exceeded1s budget. Cap in-flight retry to remaining shared budget. Preserve legitimate600-900s gallery allowances under a900s hard ceiling; the3000s stress header cannot exceed the40min job.18 runner tests and19 wrapper tests pass, retry shell check passes.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
Workflow execution regression fails over3d29c08d: head reports a bounded file timeout while the old base runner records a late named failure, so guard incorrectly charges head. Executes the actual head and base workflow scripts with fake Bun and a stale base runner.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
A retry stopped only by the global budget is inconclusive; it must not manufacture a new file-level timeout and misattribute an existing named base failure. The JSON-verdict regression fails with the current strict-budget runner.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
Package head runner+parser beside its results so base replay uses identical timeout policy with base tests/Bun. Workflow-execution regression failed ataf0430b3 over a stale base runner; now both record the same bounded file failure. Preserve a first named assertion when shared-budget retry is incomplete.18 runner and20 wrapper/workflow tests pass; actionlint and retry shell test pass.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
The deterministic delayed header lookup consumes the shared budget;2fc24eb8 still starts another Bun retry. Calls regression fails before boundary fix.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
Deterministic regression atad8f2835 shows a header lookup consuming the remaining budget still starts a retry. Re-read time after lookup and keep its original failure when no time remains.19 runner tests pass26.086s; retry shell, syntax and diff checks pass.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4777d4f75452817a742b
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 8 of 12 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo teamleaderleo changed the title ci(webviews): cap browser file timeouts and serial retry time [cx-vte03] ci(webviews): cap browser file timeouts and serial retry time Oct 26, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex xhigh): SHIP at 1736adf

Independent session nx-ci-review-browser-deadlines (01a17981-c863-7580-aa4b-2b77f68b5779, gpt-6.1-sol xhigh through acpmux) inspected the integrated four-file diff and repaired deltas. Its initial P1 about mixed head/base policies is fixed. At the final head it reran 19 runner tests (25.533s), 20 wrapper/workflow tests (20.151s), retry shell test, syntax, diff, actionlint and prepush gate: all passed, worktree clean.

Primary verdict: SHIP at the same exact head. I independently inspected the final workflow/script diff, checked test failure evidence and the clean worktree. All selected files still execute; timeouts are named failures, budget-incomplete retries retain their first real named failures, and base replay uses identical runner/parser policy with base tests and Bun. No new job, step, trigger or credential privilege. #24438 overlaps only the lint-step file and was notified; its diff is disjoint.

No full browser or native execution is claimed by this review. The remaining stop condition is browser-shard completion on merged base and two webview PRs, plus #24469's separately owned claimPaint repair.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Exact-head non-green classification at 1736adf93024567e0c596b9df56360c48f30a1ab: both completed failures are infrastructure status writers, with no test/source failure.

  • high-signal review link: PATCH .../issues/comments/... failed with API rate limit exceeded for installation (HTTP403).
  • post cmux-next/fast-gates: posting the status failed with the same installation quota error. The root fast gates job in that run passed.

Risk evidence and merge freeze are green; remaining checks were pending at the bounded snapshot. No rerun/dispatch was started to work around quota. Independent xhigh SHIP and primary review are recorded above at this exact head.

@teamleaderleo teamleaderleo added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 26, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 6 of 10 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, correctness first): SHIP at 1736adf

Independent Codex xhigh review also returned SHIP at this exact head. The completed Web tests (2/4) red is a base/infrastructure finding (cmux-next docs > the CLI and MCP reference is fresh against the action catalog) unrelated to the four changed paths; its census could not run because DNS failed while downloading historical artifacts. Status-writer failures are separately documented as installation-quota infrastructure reds.

@teamleaderleo
teamleaderleo merged commit b3c124c into feat-cmux-next Oct 26, 2026
150 of 154 checks passed
@teamleaderleo
teamleaderleo deleted the nx-ci-browser-shard-timeouts branch October 26, 2026 19:21
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

2 inactive deployments
artifacts — 1736adf9 Deployed Oct 26, 2026 by teamleaderleo via add the Windows daemon to the tree #59166
sentry-dsyms — 1736adf9 Deployed Oct 26, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #59166
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant