ci: attribution scores a lone suspect and reports app-host crashes apart

[teamleaderleo]

The main-regression attribution bot told #14922 that it broke RecoverableMainWindowLifecycleTests/closingRecoveredWindowUsesNormalCloseFinalization() in main run 36307768440. #14922 changed two localized values and two web tests. The test did not regress: the app host aborted under it (Cannot form weak reference to instance ... NSKVONotifying_NSWindow, fixed by #14925), xcodebuild recorded the test in flight as failed, and the same crash had hit every full-suite run since 02:04Z under a different test each time.

Three changes to scripts/ci/main_regression_attribution.py:

• The only pull request in the range is scored like any other. It is named only when its diff edits the failing suite (test_impact.py) or changes code or strings the suite names (reverse_test_impact.py). At score 0 the failure is listed as unattributed with the reason: the lone PR could not be diffed, changes nothing the app host loads, or does not reach the suite. A changed .xcstrings entry now counts as a changed string: its key and changed values are followed like a changed Swift literal (reverse_test_impact.literal_suites). Keep agent brand names untranslated; tighten changelog and env-rejection tests #14922's values (Codex, OpenCode) are under the 8-character minimum, so running rank_inputs on its merge commit scores 0 and gives "the only pull request in the range does not reach this suite".
• A crash victim is not a regression of the test. In a batch the accounting marked incomplete app-host run: app host restarted, only the test in flight at the restart is the crash's: it started in the live log and never finished. Swift Testing display names are mapped to functions from the cmuxTests/ sources. When the log cannot name that test, the batch's failures with no failed line of their own count as the crash's only if there are exactly as many as unexplained restarts. Every other failure in the batch stays an ordinary regression. The victim is reported as "the app host crashed while running it". The crash signature is the objc, Swift Fatal error or uncaught-exception message above the backtracer's Program crashed header, with addresses masked. When the log has no message, the line names the shard's cmux-app-host-diagnostics-* artifact instead. A PR hears about a victim only when its diff reaches the suite.
• A recurring crash is not new. The attribution now keeps the restarts of the baseline and of runs it skipped on the way. A current crash is pre-existing only when those runs already showed every one of its signatures. A crash with no signature, or with any signature no earlier run showed, is new. A recurring crash is listed once per signature in the issue, as not new, pings nobody and stays out of the bisect data. A new crash's victims go into the bisect data with "crash": true, so a crash from a PR the ranking missed still gets bisected; main_regression_bisect.py queues them like any other entry.

The PR comment marker, the per-report caps and the DATA_PREFIX payload are unchanged; the payload's tests gains the optional crash flag. workflow_guard_groups.py routes the three scripts the attribution test now loads to the ci guard group.

Verification

Current catch-up head 5c1c0ee93bc6f3c2b32d64fbad70666a853bcaf8 passes 78 attribution/bisect tests, 31 reverse-impact tests and 66 local guard steps. Independent review found no concrete correctness defects. All required current-head checks passed (CI run). Catching up with main resolves the earlier stale registry failure; no new registry entry was needed.

• python3 tests/test_ci_main_regression_attribution.py: 41 passed. New cases replay the Keep agent brand names untranslated; tighten changelog and env-rejection tests #14922 shape from a cut-down copy of the shard 7 log: a lone PR outside the suite, the victim of a host restart, and a crash the baseline also had. The issue section lists the crash once as not new, keeps the victim out of the table and bisect data, and plans no comment. Other new cases cover a crashed batch with a plain failure (the plain one is attributed and bisected), display-name mapping, the strict recurrence rule, the signature fallbacks, the artifact pointer, a new crash that pings only a PR that edits the suite, and the string catalog reach.
• python3 tests/test_ci_main_regression_bisect.py (37, including a flagged crash entry being queued) and python3 tests/test_ci_reverse_test_impact.py (31) pass.
• Parsing the real job log (108588769348) gives the victim above, the signature Cannot form weak reference to instance (0x*) of class NSKVONotifying_NSWindow and the artifact cmux-app-host-diagnostics-shard-7-run-1.

🤖 Generated with Claude Code

Changelog

none

Summary by CodeRabbit

• New Features
  • CI reports now identify app-host crashes, the test in progress, crash signatures, and related diagnostic artifacts.
  • Crash details and artifact links are included in issue sections and pull-request comments.
• Improvements
  • Recurring crashes are recognized by matching current signatures against earlier crashes. Previously seen crashes are kept separate from new regressions.
  • Pull-request attribution now considers whether changes reach the affected test suite, including through localized strings. Irrelevant changes are left unattributed.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: e8e2e0bc-569a-4565-a3b3-d9e56ac16b64

📥 Commits

Reviewing files that changed from the base of the PR and between ee2cda0 and 5c1c0ee.

📒 Files selected for processing (5)

• scripts/ci/main_regression_attribution.py
• scripts/ci/reverse_test_impact.py
• scripts/ci/workflow_guard_groups.py
• tests/test_ci_main_regression_attribution.py
• tests/test_ci_main_regression_bisect.py

 _______________________________________________
< Caches are bugs waiting to happen. - Rob Pike >
 -----------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[teamleaderleo]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[teamleaderleo]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[github-actions]

Merge receipt for 5c1c0ee93b: every check was green at merge (13 verified; 17 skipped by policy). Full suite runs on main after merge.