ci: run the suites that mention an app-source change

[teamleaderleo]

A pull request that changes app code but no cmuxTests/ file runs no app-host behavior test (hq#603). This morning #14044, #12822 and #13055 each passed CI and broke suites they never ran.

choose_ci_suite.py now takes the app diff (--app-diff-from, written by the changes job) and adds the suites reverse_test_impact.py names (#14367, #14387) to the changed-suites run:

• Only adds. No readable diff, or any selector error, adds nothing and the run routes exactly as before.
• Stays one worker. Suites are added only while the 10 min changed-suites budget has room beside the suites the PR edited. A suite that would overflow it, or that would leave the changed-suites lane, is dropped. It never turns the run into seven shards.
• Skips suites with known-main failures. A known failure that happens to pass fails a changed-suites run, which is right for a suite the PR edited and wrong for one it only reached.
• Only real batch suites. The selector also names helper types in cmuxTests/, and a selector that matches no test fails the run, so only suites discover_selectors finds are added. None whose addition changes unit_strict_steps is added, since strict steps run outside the budget.
• No compile-reuse or admission cost. A run the selector alone asked for is a canary: a re-push of already admitted inputs reuses the build and drops these suites, as the consumer canary does. Reached suites take the changed-suites worker, never compile admission.
• The report-only job stays, for measuring recall against main's failures.

Replay on #12822's tree: its 4 edited suites plus reached suites, 373 in all, 9.1 min of measured test time, no strict steps, off admission. Selection takes about 10 s in changes.

Tests: tests/test_ci_change_areas.py gains test_an_app_source_diff_runs_the_suites_that_mention_what_it_changed, committed first (fails: the flag does not exist), then the fix. test_ci_change_areas.py, test_ci_reverse_test_impact.py and test_ci_cli_product_routing.py pass locally. test_ci_main_full_suite.py has one failure that reproduces on main without this change.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

A PR that changes app source without touching cmuxTests/ used to run no app-host behavior test (hq#603); #14044, #12822 and #13055 each passed CI and broke suites they never ran. choose_ci_suite.py now takes the app diff from the changes job and adds the suites reverse_test_impact.py names to the changed-suites run.

• Only adds: a missing or unreadable diff adds nothing, and routing is unchanged.
• Stays one worker: suites are added only while the 10-minute changed-suites budget has room beside the edited suites; anything that would overflow it or leave the changed-suites lane is dropped.
• Skips suites with known-main failures, since a known failure that passes fails a changed-suites run, which is right for an edited suite and wrong for one it only reached.
• Adds only real batch suites: the selector also names helper types in cmuxTests/, and a selector that matches no test fails the run.
• Reached suites ride on a compile the run pays for and take the changed-suites worker, so admission cost is unchanged.
• A consumer edit that also changes app source keeps both the consumer canary and reached suites.
• The report-only job stays to measure recall against main's failures. On Fix Codex writer ownership during quit and restore #12822's tree the edited suites plus the selector's picks run 373 suites in 9.1 minutes; selection takes about 10 seconds.

^{Written for commit 3f610dc. Summary will update on new commits.}

Summary by CodeRabbit

• Chores
  • CI now considers changes to app code when selecting relevant test suites, while respecting suite limits and existing test-status constraints.
  • Change-impact reports now provide a fuller account of the changes considered and the estimated cost.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: c0296885-c36e-4080-bb64-db60d88f1a00

📥 Commits

Reviewing files that changed from the base of the PR and between a855dbf and 3f610dc.

📒 Files selected for processing (3)

• .github/workflows/ci.yml
• scripts/ci/choose_ci_suite.py
• tests/test_ci_change_areas.py

 _________________________________________________________________________________
< CodeRabbit is a psyop to make you think your code is being reviewed by a human. >
 ---------------------------------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}