Follow-up: finish Agent notification failure diagnostics

[austinywang]

Follow-up to #12546

PR #12546 was merged before its reusable app-host check completed. This follow-up keeps #12532 open and carries the remaining fixes made afterward:

• Trigger Agent notification semantics when scripts/ci/classify-app-host-test-output.py changes, so classifier fixes are exercised by the exact workflow.
• Keep real *** Signal <number> and contextual SIGABRT/SIGSEGV crash markers while ignoring incidental build text such as Build description signature.
• Cover signal collisions and the extracted selected-suite workflow block, including durable per-suite/aggregate logs, positive-summary success, and compile-failure red status.

Evidence

• Same-SHA historical workflow 34733959463 (8f6fc5c8f9): attempt 1 app-host job 103661975255 failed after 76/5/4 tests with the pre-Fix Agent notification semantics CI reliability #12455 Claude fixture reporting status=0, timedOut=true; attempt 2 app-host job 103664897630 passed all five selected suites (76/5/4/9/3) on the same SHA and Xcode 26.3/macOS 15 arm64 toolchain. This proves the old waitUntilExit() observer race.
• Current-HEAD 34794477469 attempt 1 (720f24ade3), app-host job 103824957050: The test runner timed out while preparing to run tests; diagnosis pre-test app-host failure, executed_tests=0, no semantic assertions.
• Current-HEAD 34794477469 attempt 2 (720f24ade3), app-host job 103829129669: the same pre-test app-host timeout on a different WarpBuild arm64 machine; diagnosis pre-test app-host failure, executed_tests=0, no semantic assertions.
• Post-Fix Agent notification semantics CI reliability #12455 controls 34749372115 and 34750823146 execute all five selected suites successfully.

The current repeated failures are hosted runner/testmanagerd setup defects, not semantic assertion failures. The workflow now preserves and reports that distinction and remains red when assertions never execute. The follow-up changes no notification product semantics and does not remove or weaken Claude/Codex semantic coverage.

Related to #12532

• Same-SHA attempt 3 34794477469 attempt 3, app-host job 103833720547: the same The test runner timed out while preparing to run tests failure after ~25 minutes, with executed_tests=0 and no semantic assertions. Three attempts on the same final SHA now agree on the pre-test app-host category.

Handoff: the remaining red check belongs to the hosted XCTest/testmanagerd runner path used by the reusable workflow. Product notification semantics and package tests are green; no further notification code change is justified by these zero-test failures.

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: cec2e938-bda9-416c-801d-a59e48574533

📥 Commits

Reviewing files that changed from the base of the PR and between 97f8a15 and 720f24a.

📒 Files selected for processing (3)

• .github/workflows/agent-notification-tests.yml
• scripts/ci/classify-app-host-test-output.py
• tests/test_ci_app_host_test_output.py

Included review availability: Your plan provides up to 10 included reviews per hour; 2 remain after this review.

---

📝 Walkthrough

Walkthrough

The app-host classifier now detects numeric and named signal failures. Tests cover diagnosis edge cases, selected-suite outcomes, and log retention. The notification test workflow also runs when the classifier changes.

Changes

App-host failure classification

Layer / File(s)	Summary
App-host failure detection and diagnosis scripts/ci/classify-app-host-test-output.py	The classifier detects numeric and named signal messages in addition to existing app-host failure patterns.
Classification validation and workflow coverage tests/test_ci_app_host_test_output.py, .github/workflows/agent-notification-tests.yml	Tests cover build-signature text, signal-based pre-test failures, selected-suite success and compile failure, and log retention. The workflow path filter includes the classifier script.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 720f2

The updated classifier recognizes the supported contextual signal crash output, with no remaining actionable merge risk identified.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 2 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (24 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Swift Actor Isolation	✅ Passed	PASS: The authoritative diff changes only one YAML workflow and two Python files. It adds no production Swift files or Swift actor-isolation declarations. The actor-isolation failure conditions theref…
Cmux Swift Blocking Runtime	✅ Passed	PASS: The authoritative PR diff changes only one YAML workflow, one Python classifier, and one Python test file. It contains no changed .swift paths and introduces no Swift blocking or timing-based …
Cmux Browser Automation Off-Main	✅ Passed	The pull request changes only a GitHub workflow trigger, a Python app-host output classifier, and its Python tests. The authoritative diff contains no browser socket automation commands, WebKit/AppKit…
Cmux Expensive Synchronous Load	✅ Passed	PASS — the pull request changes only one GitHub workflow, one Python classifier, and Python tests. The authoritative diff contains no .swift files or production Swift changes. It adds no synchronous…
Cmux Cache Substitution Correctness	✅ Passed	PASS: The reviewed diff changes one GitHub Actions workflow, one Python classifier, and one Python test file. It does not change production Swift, TypeScript, or JavaScript code. Therefore, the cache-…
Cmux No Hacky Sleeps	✅ Passed	PASS. The pull request adds no fixed sleep, timer, polling loop, backoff, or wall-clock wait. The production Python change only updates signal-matching regular expressions and diagnosis selection. The…
Cmux Algorithmic Complexity	✅ Passed	The pull request does not introduce an algorithmic-complexity violation. The production changes are limited to a Python regular expression and one additional matcher in the existing single-pass log-li…
Cmux Swift Concurrency	✅ Passed	PASS. The pull request changes only one YAML file and two Python files. The authoritative diff contains no Swift paths and no added Swift concurrency patterns. Therefore, the Swift concurrency check i…
Cmux Swift @Concurrent	✅ Passed	PASS: The authoritative PR diff changes only .github/workflows/agent-notification-tests.yml, scripts/ci/classify-app-host-test-output.py, and tests/test_ci_app_host_test_output.py. It adds no Sw…
Cmux Swift Package Boundaries	✅ Passed	PASS. The authoritative diff changes only one YAML workflow and two Python files. It contains no Swift source, Package.swift, or SwiftPM target changes. Therefore, the Swift package boundary rule is n…
Cmux Swiftpm Lockfiles	✅ Passed	PASS. The authoritative PR diff changes only .github/workflows/agent-notification-tests.yml, scripts/ci/classify-app-host-test-output.py, and its tests. The workflow change adds a pull-request pat…
Cmux Swift Logging	✅ Passed	PASS: The pull request changes only YAML, Python, and Python test files. It adds no production Swift logging or materially changes any Swift logging statement. The added print calls are inside a tes…
Cmux User-Facing Error Privacy	✅ Passed	PASS: The diff changes a CI workflow, a CI-only classifier, and its tests. The new diagnostic text uses generic categories and signal markers such as Signal 5 and SIGABRT; it does not add vendor n…
Cmux Full Internationalization	✅ Passed	PASS. The pull request changes only a GitHub Actions trigger, CI diagnostic Python, and tests. The added signal patterns and category text are operational test-output diagnostics, not Swift UI, web UI…
Cmux Swiftui State Layout	✅ Passed	PASS. The authoritative diff changes only one YAML workflow and two Python files. It adds no Swift files or SwiftUI code, and no changed lines contain ObservableObject, @Published, @Observable, Geomet…
Cmux Architecture Rethink	✅ Passed	PASS. The authoritative diff changes only one YAML workflow, one Python classifier, and one Python test file. It changes no Swift source or SwiftUI/AppKit lifecycle ownership. The patch adds a workflo…
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	PASS. The reviewed range changes only one YAML workflow and two Python files; it changes no Swift, NSWindow, NSPanel, NSWindowController, SwiftUI Window, or WindowGroup code. The auxiliary-window clos…
Cmux Source Artifacts	✅ Passed	The PR changes only three tracked source/config/test files: the workflow, the classifier script, and its Python test. No generated logs, screenshots, recordings, caches, build output, dependency check…
Cmux No Test Or Debug Seam In Production Source	✅ Passed	The pull-request diff changes only one workflow YAML file, one Python classifier, and one Python test file. It contains no changed Swift file under a production Sources/ path, so the specified produ…
Cmux No Ambient Global State	✅ Passed	PASS: The custom check applies only to production Swift changes. The authoritative PR diff changes one YAML file and two Python files, with no Swift, Objective-C, or header files. Therefore, the ambie…
Title check	✅ Passed	The title clearly identifies this as a follow-up that completes Agent notification failure diagnostics. It is concise and related to the main changes.
Description check	✅ Passed	The description explains what changed, why the follow-up was needed, and provides detailed testing evidence. It omits the template's formal Testing, Demo Video, Review Trigger, and Checklist sections,…

Full details: Docstring Coverage

Explanation

Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 2 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-12532-agent-notification-flaky

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.