ci: make a changed-suites run prove a known-failure fix

[teamleaderleo]

A PR that fixes a test listed in scripts/ci/app-host-known-failures.json currently goes green whether or not the fix works, because the app-host ratchet tolerates every listed failure. #14076 merged that way: its changed-suites lane ran the suite it was fixing, failed 4 of 4 cases, and reported success (RATCHET_KNOWN_FAILURE ... known-main failures tolerated: 1).

In a changed-suites run (CMUX_APP_HOST_UNIT_SELECTORS set), a listed test that passes now fails the run with RATCHET_KNOWN_NOW_PASSING <id> and a line saying to drop the entry. With the entry removed, the run has to pass on its own. So a fix PR carries its catalog removal, and its green check means the test passed. Main's full shards only log RATCHET_KNOWN_NOW_PASSING, because a flaky entry can pass on any single run and main shouldn't fail for that.

A PR that edits a listed suite for an unrelated reason and happens to make it pass will also be asked to drop the entry. That's intended: the catalog only shrinks, and a passing run in the PR that touched the suite is the evidence for shrinking it.

This applies to every catalog classification, including timeout/hang and unknown, because #14076's target was a timeout/hang entry. The cost: a PR whose selected suites include a flaky entry can be asked to drop it after one lucky pass. Selection also pulls in suites that only reference a changed helper, not just suites the PR edited. If that turns out noisy, the next step is to limit the hard fail to suites the PR edits directly. Suites run as strict focused gates (FOCUSED_GATE_SELECTORS) never reach check-run and aren't covered.

Validation, local (no build): python3 tests/test_ci_app_host_result_accounting.py passes, including a new case covering main-shard reporting, the changed-suites failure, and the pass after removal. The 7 ratchet and batch tests in tests/test_ci_change_areas.py pass, including the one that drives run-app-host-unit-batches.sh against a fake runner. bash -n is clean on the batch script.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Makes changed-suites runs fail when a test listed in app-host-known-failures.json passes, so a fix PR's green check now proves the fix works. Previously the ratchet tolerated every listed failure, so a fix could merge with the target test still failing.

Behavior

• Changed-suites runs now report RATCHET_KNOWN_NOW_PASSING and fail until the entry is removed from app-host-known-failures.json; the run must then pass on its own.
• Still-failing known entries keep their RATCHET_KNOWN_FAILURE lines even when a different entry passes.
• Main's full shards only report the passing entry, since a flaky entry can pass on any single run.
• A PR that makes a listed suite pass for any reason is asked to drop the entry, so the catalog only shrinks.

^{Written for commit d3ac629. Summary will update on new commits.}

Summary by CodeRabbit

• CI Improvements
  • When selected test suites change, CI flags tests that pass but remain listed as known failures, points to the entries that need updating, and fails the run until those entries are removed.
  • Runs that do not change selected suites retain the existing result-accounting behavior.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0bfcb8ba-8839-4b69-bafe-2d34fb96df23

📥 Commits

Reviewing files that changed from the base of the PR and between 5887891 and fb9c296.

📒 Files selected for processing (3)

• scripts/ci/app_host_result_accounting.py
• scripts/ci/run-app-host-unit-batches.sh
• tests/test_ci_app_host_result_accounting.py

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

The accounting command now detects known failures that pass. For changed suites, it reports those tests and fails the run until their entries are removed from the known-failures catalog. The batch runner enables this check when suite selectors are set.

Changes

Changed-suite known-failure accounting

Layer / File(s)	Summary
Changed-suite accounting flow scripts/ci/app_host_result_accounting.py, scripts/ci/run-app-host-unit-batches.sh, tests/test_ci_app_host_result_accounting.py	check_run accepts a changed_suites option and reports passing catalog-known tests. When the option is enabled and such tests exist, the run fails and reports the catalog-removal diagnostic. The batch runner enables the option when suite selectors are set. Tests cover changed and unchanged suites.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to fb9c2

Changed-suite runs now flag passing tests that remain in the known-failures catalog. No actionable merge-blocking issue remains after normal checks.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (24 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input	✅ Passed	PASS. The pull request changes only app-host CI result accounting, a batch shell wrapper, and accounting tests. The diff adds --changed-suites handling for known test failures and does not change Cl…
Cmux Swift Actor Isolation	✅ Passed	PASS: The authoritative pull-request diff changes only Python CI accounting, a shell CI script, and Python tests. It introduces no production Swift changes, so it cannot introduce or worsen Swift 6 ac…
Cmux Swift Blocking Runtime	✅ Passed	The pull request changes only Python, shell, and Python test files. The authoritative diff contains no Swift files or Swift runtime code, so the Swift blocking-runtime check is not applicable.
Cmux Browser Automation Off-Main	✅ Passed	The pull request changes only Python CI accounting, a shell batch runner, and its tests. It does not modify browser socket automation, Sources/TerminalController.swift, or `ControlCommandExecutionPo…
Cmux Expensive Synchronous Load	✅ Passed	PASS: The pull request changes only two Python files and one shell script. The authoritative diff contains no Swift files or production Swift changes, so it cannot introduce an expensive synchronous S…
Cmux Cache Substitution Correctness	✅ Passed	PASS. The authoritative PR diff changes only Python, shell, and Python test files. It contains no production Swift, TypeScript, or JavaScript changes, so the cache-substitution correctness condition i…
Cmux No Hacky Sleeps	✅ Passed	The pull request introduces no fixed sleep, timer, polling, delayed dispatch, or wall-clock synchronization. The added Python logic only reports known passing tests and changes the ratchet mode. The s…
Cmux Algorithmic Complexity	✅ Passed	The diff adds one pass over the known-failure dictionary and sorts the matching identifiers in scripts/ci/app_host_result_accounting.py (O(K log K)). The catalog currently has 15 entries, and the pa…
Cmux Swift Concurrency	✅ Passed	The pull-request diff changes only Python, shell, and Python test files. It contains no Swift code, so it does not introduce or expand any listed Swift concurrency pattern.
Cmux Swift @Concurrent	✅ Passed	The pull request changes only two Python files and one shell script. It contains no Swift changes, so the Swift @concurrent check is not applicable.
Cmux Swift Package Boundaries	✅ Passed	The reviewed diff changes only Python, shell, and Python test files. It contains no production Swift changes, so the Swift package-boundary check does not apply.
Cmux Swiftpm Lockfiles	✅ Passed	The PR changes only Python CI accounting, a shell batch runner, and a Python test. The authoritative diff contains no Package.swift, Package.resolved, Xcode project/workspace, .gitignore, workflow, or…
Cmux Swift Logging	✅ Passed	PASS: The pull request changes only Python, shell, and Python test files. The authoritative diff contains no Swift files or Swift production code, so the cmux Swift logging check is not applicable.
Cmux User-Facing Error Privacy	✅ Passed	PASS. The changed output is confined to scripts/ci/app_host_result_accounting.py and scripts/ci/run-app-host-unit-batches.sh, which run from GitHub Actions workflows for internal CI. The messages …
Cmux Full Internationalization	✅ Passed	PASS. The pull request changes only CI accounting, a shell batch runner, and tests. The added text is CI diagnostic/help output and developer-facing test text, not Swift UI, app catalog content, web U…
Cmux Swiftui State Layout	✅ Passed	The pull request changes only Python CI accounting, a shell script, and a Python test. The authoritative diff contains no Swift or SwiftUI files and no SwiftUI state, GeometryReader, lazy/list row, or…
Cmux Architecture Rethink	✅ Passed	PASS: The pull request changes only Python CI accounting, a shell CI wrapper, and Python tests. The authoritative diff contains no Swift or SwiftUI/AppKit code, so it does not introduce or expand any …
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	The pull request changes only scripts/ci/app_host_result_accounting.py, scripts/ci/run-app-host-unit-batches.sh, and tests/test_ci_app_host_result_accounting.py. The authoritative diff contains …
Cmux Source Artifacts	✅ Passed	The PR changes only intentional CI source, shell orchestration, and a focused Python test: scripts/ci/app_host_result_accounting.py, scripts/ci/run-app-host-unit-batches.sh, and `tests/test_ci_app…
Cmux No Test Or Debug Seam In Production Source	✅ Passed	The pull request changes only Python, shell, and Python test files. It adds no Swift file under a production Sources/ path, so it cannot introduce a test or debug seam covered by this check.
Title check	✅ Passed	The title clearly and concisely describes the main change: changed-suites runs now verify fixes for known failures.
Description check	✅ Passed	The description explains the problem, intended behavior, scope, trade-offs, and validation commands. It is mostly complete, although it does not include the template checklist or a demo attachment for…

• Fix all pre-merge checks with AI

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}