ci: run the dedicated step when a PR edits an env-gated test

[teamleaderleo]

A PR that edits a test only a dedicated CI step can run could pass without running that test.

The five-tab renderer memory test in GhosttySurfaceOverlayTests skips itself unless its step sets CMUX_RENDERER_MEMORY_REGRESSION=1. That step ran only on shard 6. A test-only PR that edited the test got a changed-suites run of GhosttySurfaceOverlayTests inside compile admission, where the test reported "skipped" and CI went green.

strict_steps() in choose_ci_suite.py now also counts any shard step that names a suite with -only-testing:. A changed-suites run of that suite runs the step too, on the shard 8 worker. If such a step's if: cannot be selected through unit_strict_steps, routing falls back to all seven shards, so the step still runs on its own shard. The renderer memory step now accepts unit_strict_steps.

#14366 was not a case of this. Its compile admission log shows all three edited suites ran and passed: KeyboardShortcutSettingsFileStoreTests (33 XCTest cases), plus SurfacePaneFactoryFocusTests and SurfaceSelectionTests (23 Swift Testing tests), 56 typed results. Shard 8 was skipped because compile admission ran those suites itself.

Proof on this PR: comment-only edits to SurfaceSelectionTests and to the renderer memory test. This PR's CI should run SurfaceSelectionTests and the "Run five-tab renderer memory regression" step on the changed-suites worker.

Tests: tests/test_ci_change_areas.py gains a regression test that fails on main, plus one that pins #14366's routing and checks that an untraceable test edit runs every suite. actionlint is clean.

🤖 Generated with Claude Code

---

Summary by cubic

Fixes a CI gap where a PR editing a test that runs only in a dedicated CI step could pass without executing that test.

The five-tab renderer memory test skips itself unless its step sets CMUX_RENDERER_MEMORY_REGRESSION=1, but that step only ran on shard 6. A test-only PR editing the test got a changed-suites run where it reported "skipped" and CI went green.

• strict_steps() now counts any shard step that names a suite with -only-testing:, quoted or unquoted, so a changed-suites run of that suite also runs the step on shard 8.
• If a step that must run can't be selected through unit_strict_steps, routing falls back to all seven shards so the step still runs on its own shard.
• The renderer memory step now accepts unit_strict_steps.
• Adds regression tests: one pinning that test: fix three app-host tests failing on main #14366 was not affected by this gap, one proving a looped -only-testing:$suite step lists only strict suites, and one covering the unselectable-step fallback.

^{Written for commit 3d628ac. Summary will update on new commits.}

Summary by CodeRabbit

• Improvements

  • CI more reliably routes test changes to the relevant test suites and dedicated checks, including the renderer memory regression check.
  • Changes that cannot be attributed to a specific suite continue to receive broader test coverage.

• Tests

  • Added coverage for test-change routing, dedicated checks, and fallback behavior when changes cannot be matched to a suite.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: bfe4db46-a2de-43ef-a520-78f448127710

📥 Commits

Reviewing files that changed from the base of the PR and between 1036ca8 and 3d628ac.

📒 Files selected for processing (2)

• scripts/ci/choose_ci_suite.py
• tests/test_ci_change_areas.py

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

CI suite selection maps changed test suites to eligible strict workflow steps. The renderer-memory regression step also runs when changed-suite routing selects it.

Changes

Changed-suite CI routing

Layer / File(s)	Summary
Suite attribution and strict-step selection scripts/ci/choose_ci_suite.py, tests/test_ci_change_areas.py, cmuxTests/SurfaceSelectionTests.swift, cmuxTests/TerminalAndGhosttyTests.swift	The routing logic maps test suites to owning strict steps that can be selected through inputs.unit_strict_steps. Tests cover suite attribution and selection outcomes. Comments identify suites that use this routing.
Regression step selection .github/workflows/ci-macos.yml	The renderer-memory regression step runs for its focused-regression shard or when selected through unit_strict_steps.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 3d628

Changed suites can select their owning CI steps, and uncertain ownership broadens testing to all seven shards. No concrete coverage regression remains, so the PR is mergeable subject to normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 3d628

The change closes a path where CI could pass after skipping an edited test. No new security exposure was identified, but the assessment does not include a live CI run.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The identified security-relevant outcome is CI coverage for PR-edited tests. The flagged public entrypoint is a test-only caller, with no identified expansion of a production service or authorization boundary.

Trust Boundaries and Controls

• inferred — A PR's changed-file information influences test selection, but an unselectable required step broadens execution to the full shard set rather than allowing a narrowed run to omit it.

Resilience and Maintainability Implications

• observed — The fallback regression removes the step's selection condition and verifies that suite narrowing is abandoned, preserving the broader execution path.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (24 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main CI routing change for environment-gated tests.
Description check	✅ Passed	The description clearly explains the CI gap, the routing fix, the fallback behavior, and the added regression tests. It provides testing information, although it does not use the template's separate T…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input	✅ Passed	PASS: The PR changes macOS CI routing, test comments, and choose_ci_suite.py plus routing tests. The diff adds no Cloud terminal creation, persistent cmux-tui transport, manual-pane admission, att…
Cmux Swift Actor Isolation	✅ Passed	The pull request changes only comments in two Swift test files. It introduces no production Swift declarations, actor-isolation changes, Sendable reference types, or background UI-store access. The re…
Cmux Swift Blocking Runtime	✅ Passed	PASS. The pull request changes only comments in the two Swift test files. The added Swift lines introduce no semaphores, waits, sleeps, delayed dispatch, polling, main-queue sync, or manual locks. The…
Cmux Browser Automation Off-Main	✅ Passed	The PR does not change browser socket automation routing. The rule-scoped files Sources/TerminalController.swift and ControlCommandExecutionPolicy.swift are unchanged, and no added or removed line…
Cmux Expensive Synchronous Load	✅ Passed	PASS: The PR does not add or move an expensive synchronous agent-history load. The only Swift changes are comments in cmuxTests/SurfaceSelectionTests.swift and `cmuxTests/TerminalAndGhosttyTests.swi…
Cmux Cache Substitution Correctness	✅ Passed	PASS: The pull request does not introduce a cache substitution. Its Swift changes only add comments; the other changes are Python, YAML, and test code. No production Swift, TypeScript, or JavaScript p…
Cmux No Hacky Sleeps	✅ Passed	PASS. The PR adds no fixed sleeps, timers, polling, delayed dispatch, or wall-clock synchronization in covered runtime scripts. The Python changes only route CI suites, and the added Python tests are …
Cmux Algorithmic Complexity	✅ Passed	The changed production code only extends strict_steps() to inspect CI workflow configuration. Its nested scan covers 54 fixed workflow steps and 38 fixed focused selectors, not user-owned records or…
Cmux Swift Concurrency	✅ Passed	PASS: The only changed Swift lines are comments in SurfaceSelectionTests.swift and TerminalAndGhosttyTests.swift. They add no Dispatch, Combine, completion-handler, or fire-and-forget Task cod…
Cmux Swift @Concurrent	✅ Passed	PASS: The Swift diff adds only comments in SurfaceSelectionTests.swift and TerminalAndGhosttyTests.swift. It does not add or modify async, nonisolated, @concurrent, actor isolation, or async…
Cmux Swift Package Boundaries	✅ Passed	The PR changes only comments in two Swift test files. It introduces no production Swift logic, app-target feature, or reusable domain code. The Swift package-boundary rule is therefore not triggered.
Cmux Swiftpm Lockfiles	✅ Passed	PASS. The PR changes CI routing, Swift test comments, and Python routing tests only. The authoritative diff contains no Package.swift, Package.resolved, .gitignore, or Xcode project package-reference …
Cmux Swift Logging	✅ Passed	PASS. The only changed Swift lines are comments in cmuxTests test files. They add no print, debugPrint, dump, NSLog, file logging, Logger declaration, or sensitive-data logging. The logging rule permi…
Cmux User-Facing Error Privacy	✅ Passed	PASS: The diff changes only GitHub Actions CI routing, a CI selector script, tests, and developer-only comments. The script output is consumed as GitHub Actions job outputs, and no changed text reache…
Cmux Full Internationalization	✅ Passed	PASS. The diff changes only CI workflow logic, CI tooling, tests, and developer-only comments. It adds no production user-facing text, app localization catalog entries, or web locale/message files. Th…
Cmux Swiftui State Layout	✅ Passed	PASS: The pull request does not introduce SwiftUI changes. The only Swift changes add comments in test files; the remaining changes are CI and Python logic. The diff adds no ObservableObject, @Publish…
Cmux Architecture Rethink	✅ Passed	PASS: The pull request does not introduce a Swift architecture change. The only Swift additions are comments in two test files. The executable changes are CI workflow, Python routing, and Python tests…
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	PASS — The only Swift changes add comments in cmuxTests/SurfaceSelectionTests.swift and cmuxTests/TerminalAndGhosttyTests.swift. The diff adds no NSWindow, NSPanel, NSWindowController, SwiftUI Win…
Cmux Source Artifacts	✅ Passed	The diff changes only tracked workflow configuration, Swift tests, and Python CI/test scripts. No logs, media, caches, build output, temporary directories, dependency checkouts, or other artifact-like…
Cmux No Test Or Debug Seam In Production Source	✅ Passed	The PR changes no Swift file under a production Sources/ path. The only changed Swift files are under cmuxTests/, and their changes are comments. Therefore, the production test/debug seam rule doe…

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}