Skip to content

fix(latency): report incomplete browser engine coverage - #27674

Closed
teamleaderleo wants to merge 1 commit into
feat-cmux-nextfrom
lane/latency-engine-coverage-5179
Closed

teamleaderleo wants to merge 1 commit into
feat-cmux-nextfrom
lane/latency-engine-coverage-5179

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes manaflow-ai/cmuxterm-hq#5179.

The latency scoreboard used to catch every browser launch error as an unstructured warning and continue. A partial Chromium/WebKit sweep could therefore look complete, including the universal “every action fits one frame” line.

This change:

  • records the selected engine set before launching browsers;
  • preserves each launch failure as { engine, error, stage: "launch" };
  • writes expected/measured engine coverage and launch failures to JSON and the GitHub summary;
  • suppresses the universal success line when engine coverage is incomplete; and
  • keeps the nonblocking --ci scoreboard and explicit --engine filtering unchanged.

The launch-selection helper has focused tests for complete coverage, one-engine failure, both-engine failure, and explicit one-engine selection. No latency samples, budgets, or runtime exit policy changed.

Performance

No perf numbers: this changes latency evidence reporting only; it does not alter measured latency, budgets, or runtime performance.

Validation will run through the managed Linux webview test lane.

🤖 Generated with Claude Code
Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes manaflow-ai/cmuxterm-hq#5179.

The latency scoreboard previously caught browser launch errors as unstructured warnings and continued, so a partial Chromium/WebKit sweep could look complete and even print "every action fits one frame."

This change records the selected engine set before launching, keeps each launch failure as structured { engine, error, stage } data, and writes expected/measured engine coverage plus launch failures to the JSON output and GitHub summary. The universal success line is suppressed when coverage is incomplete, replaced by a measured/expected engine count. The run now exits non-zero when engine launch coverage is incomplete (unless --no-fail), matching the existing action-budget exit. Nonblocking --ci scoreboard behavior and explicit --engine filtering (including an invalid filter) are unchanged.

The new launch-selection helper has focused tests for complete coverage, one-engine failure, both-engine failure, and explicit one-engine selection. No latency samples or budgets changed.

Written for commit 6b0ed8c. Summary will update on new commits.

View guided diff Turn on auto-fix

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 6b0ed8c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27674 -f ref=lane/latency-engine-coverage-5179 -f platform=macos -f backend_mode=local

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026
@teamleaderleo
teamleaderleo force-pushed the lane/latency-engine-coverage-5179 branch 2 times, most recently from 195f0d3 to e726aa2 Compare October 29, 2026 23:45
@cursor

cursor Bot commented Oct 29, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

Record selected engines and structured launch failures so partial latency sweeps cannot look complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-8653e9b405a501fdfdc3
@teamleaderleo
teamleaderleo force-pushed the lane/latency-engine-coverage-5179 branch from e726aa2 to 6b0ed8c Compare October 29, 2026 23:52
@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 6b0ed8c

No blocking findings.

Nonblocking note: The PR description says runtime exit policy is unchanged, but webviews/scripts/latency/run.ts:261 now exits nonzero for partial engine launch failures unless --ci or --no-fail is set.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 6b0ed8c: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Superseded by #28156, a backlog-drain carry that keeps this PR's commits and their authorship and resolves the base merge conflict. Closing this one in its favour.

teamleaderleo added a commit that referenced this pull request Oct 31, 2026
* fix(latency): report browser engine launch coverage

Record selected engines and structured launch failures so partial latency sweeps cannot look complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-8653e9b405a501fdfdc3

* fix(latency): preserve schema and fail incomplete engine runs

Agent-Session: https://cmux.com/deeplink/agent?id=aid-de6a18975ad3ec68b1b1

* test(latency): cover chunked server readiness

Agent-Session: https://cmux.com/deeplink/agent?id=aid-01581fe5da773d909472

* test(latency): server readiness strips colour escapes around the port

Keeps #28494's SGR-tolerant banner match covered after folding it into waitForServerReady.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-159afad10a5ab45ea16c

* style(latency): oxfmt the engine-selection import and test

web lint on the merge failed oxfmt --check for these two files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-159afad10a5ab45ea16c

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant