ci: register the nine unregistered Python tests by teamleaderleo · Pull Request #13712 · manaflow-ai/cmux

teamleaderleo · 2026-09-22T17:00:46Z

validate_test_execution_registry.py fails on main. Nine tests exist with no registry entry, which fails workflow-guard-tests / preflight, then guards / Guard status, then ci-status — on every open pull request, regardless of what the PR changes.

Reproduced on unmodified main (1847ef0621):

Python test execution registry validation failed:
  - tests/test_ci_actionlint_covers_every_workflow.py: test exists but has no execution registry entry
  - tests/test_ci_app_host_result_accounting.py: ...
  - tests/test_ci_selective_layer_wiring.py: ...
  - tests/test_ci_workflow_guards_are_wired.py: ...
  - tests/test_ios_screenshot_capture_guard.py: ...
  - tests/test_ios_upload_array_expansion.py: ...
  - tests/test_merge_xcstrings.py: ...
  - tests/test_release_homebrew_gate.py: ...
  - tests/test_tui_publish_dispatch_budget.py: ...

Each test arrived with its own PR and none added the entry the guard requires.

How each was classified

I checked where every one actually runs rather than assigning a lane by shape.

Lane	Tests	Why
`linux-guard`	7	invoked from `ci-guards.yml`, which is what that lane requires
`legacy`	2	run in `testbox-broker-guard.yml` and `ci-artifact-transport.yml`; the registry models no lane for those, and the other tests those workflows run are already `legacy`

tests/test_ios_screenshot_capture_guard.py ran nowhere — no workflow invoked it. It is a static guard over the fastlane screenshot lane's config and passes on Linux with no iOS tooling, so this wires it into ci-guards.yml beside the other release-ios guards instead of inventorying a test that never executes. That is a step, not just a registry line, so it is visible in the diff.

Validation

Python test execution registry valid: 218 tests (legacy=83, linux-guard=74,
macos-cli-no-socket=48, macos-cli-no-socket-post-fish=11, macos-shell=1, macos-shell-fish=1)

tests/test_ios_screenshot_capture_guard.py and tests/test_ci_workflow_guards_are_wired.py both pass locally. Full tests/test_ci_* sweep shows no new failures against main (test_ci_change_areas.py and test_ci_sparkle_build_monotonic.sh fail identically on unmodified main here — they shell out to gh and to macOS-only tooling).

The two legacy entries are inventory, not an endorsement: modelling a real lane for tests owned by testbox-broker-guard.yml and ci-artifact-transport.yml is worth doing separately.

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by cubic

Registers the nine Python tests missing from the test execution registry so the workflow-guard-tests / preflight guard and the ci-status checks it feeds stop failing on every open PR.

Seven tests already run in ci-guards.yml and are registered as linux-guard.
Two tests run in testbox-broker-guard.yml and ci-artifact-transport.yml and are registered as legacy, matching the other tests those workflows run.
tests/test_ios_screenshot_capture_guard.py ran nowhere; it's now wired into ci-guards.yml beside the other release-ios guards instead of being inventoried as a test that never executes.

^{Written for commit 16dcac4. Summary will update on new commits.}

Summary by CodeRabbit

Tests
- Expanded automated validation for iOS release workflows, including screenshot capture failure handling.
- Added additional checks for Linux and legacy test lanes.
Chores
- Strengthened continuous integration safeguards to improve release verification consistency.

`validate_test_execution_registry.py` fails on main: nine tests exist with no registry entry, which fails workflow-guard-tests/preflight, then Guard status, then ci-status on every open pull request. The tests arrived with their own PRs; none of them added the registry entry the guard requires. Seven already run in ci-guards.yml and are registered as linux-guard. Two run in other workflows -- test_ci_actionlint_covers_every_workflow.py in testbox-broker-guard.yml and test_ci_selective_layer_wiring.py in ci-artifact-transport.yml -- and the registry has no lane for that, so they enter the legacy inventory lane the same way the other tests those two workflows run already do. tests/test_ios_screenshot_capture_guard.py ran nowhere at all. It is a static guard on the fastlane screenshot lane and passes on Linux with no iOS tooling, so wire it into ci-guards.yml beside the other release-ios guards rather than inventory a test that never executes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-09-22T17:01:09Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

coderabbitai · 2026-09-22T17:01:57Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 01ff04fb-6b83-41a4-9426-7a7cd69d6f4d

📥 Commits

Reviewing files that changed from the base of the PR and between 15a8107 and 16dcac4.

📒 Files selected for processing (2)

.github/workflows/ci-guards.yml
tests/test-execution.toml

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.

📝 Walkthrough

Walkthrough

The CI workflow now runs an iOS screenshot capture guard for release-ios. The test registry adds seven linux-guard entries and two legacy entries.

Changes

CI guard coverage

Layer / File(s)	Summary
Guard and test registry updates `.github/workflows/ci-guards.yml`, `tests/test-execution.toml`	The workflow runs `tests/test_ios_screenshot_capture_guard.py` when `matrix.group == 'release-ios'`. The registry adds seven `linux-guard` tests and two `legacy` tests without additional requirements.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 16dca

The change expands CI coverage with an iOS guard and nine registry entries, with no current evidence of production impact or merge-blocking risk.

🚥 Pre-merge checks | ✅ 25

✅ Passed checks (25 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary change: registering nine unregistered Python tests. It is concise and specific.
Description check	✅ Passed	The description clearly explains the problem, the classification of the tests, the iOS workflow wiring, and the validation performed. It does not include the template checklist, review-trigger block, …
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input	✅ Passed	PASS: The pull request only adds CI test registration entries and invokes an existing iOS screenshot guard in the `release-ios` job. The authoritative diff adds no Cloud terminal creation, cmux-tui tr…
Cmux Swift Actor Isolation	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The diff contains no Swift production changes, so it cannot introduce or worsen Swift 6 actor iso…
Cmux Swift Blocking Runtime	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds Python test registration and a Python CI step. The diff contains no Swift changes, so the…
Cmux Browser Automation Off-Main	✅ Passed	PASS: The PR changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It does not change `Sources/TerminalController.swift`, the browser command policy, worker routing, or polic…
Cmux Expensive Synchronous Load	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds CI test registration and a Python test step. It adds no production Swift code or synchron…
Cmux Cache Substitution Correctness	✅ Passed	PASS: The review-scoped diff changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds Python test registration and a CI step; it does not change production Swift, TypeSc…
Cmux No Hacky Sleeps	✅ Passed	PASS. The PR changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The added workflow command runs a Python test, and the rule explicitly excludes GitHub Actions YAML. The re…
Cmux Algorithmic Complexity	✅ Passed	PASS: The authoritative PR diff changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds a Python test invocation and nine test registry entries. It introduces no produc…
Cmux Swift Concurrency	✅ Passed	PASS — The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds a Python CI step and test registry entries. It does not change cmux-owned Swift code or …
Cmux Swift `@Concurrent`	✅ Passed	The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The authoritative diff contains no Swift files or Swift code, so it cannot introduce any `@concurrent` …
Cmux Swift Package Boundaries	✅ Passed	The check applies only to production Swift changes. The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`; the authoritative diff contains no Swift or package…
Cmux Swiftpm Lockfiles	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The diff adds a CI test step and registry entries. It does not change `Package.swift`, `Package.r…
Cmux Swift Logging	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds a CI test step and nine Python test registry entries. It adds or materially changes no pr…
Cmux User-Facing Error Privacy	✅ Passed	PASS: The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds CI test execution and registry entries. These are internal CI/test changes, not productio…
Cmux Full Internationalization	✅ Passed	PASS. The pull request changes only CI workflow wiring and test-execution registry entries. The added text is test names, workflow step labels, lane identifiers, and paths. These are operational/test …
Cmux Swiftui State Layout	✅ Passed	The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The diff adds CI test registration and a Python test step. It contains no SwiftUI, `ObservableObject`, …
Cmux Architecture Rethink	✅ Passed	The check is not applicable. The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`; it changes no Swift files and adds no architectural trigger such as sleeps…
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	The pull request changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. The diff contains no Swift, NSWindow, NSPanel, NSWindowController, SwiftUI Window, or WindowGroup chang…
Cmux Source Artifacts	✅ Passed	PASS. The diff changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds a CI test step and nine test-registry entries, which are intentional test-system configuration. B…
Cmux No Test Or Debug Seam In Production Source	✅ Passed	PASS. The authoritative PR diff changes only `.github/workflows/ci-guards.yml` and `tests/test-execution.toml`. It adds CI wiring and registry entries, with no Swift file under a production `Sources/`…

✨ Finishing Touches

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

teamleaderleo · 2026-09-22T17:15:56Z

This duplicates #13710, opened five minutes earlier, and #13710 is the better of the two to keep.

Both register the same nine paths. The difference is the lane they give two of them:

	#13710	this PR
`tests/test_ci_actionlint_covers_every_workflow.py`	`linux-guard`	`legacy`
`tests/test_ci_selective_layer_wiring.py`	`linux-guard`	`legacy`
other seven	`linux-guard`	`linux-guard`

legacy understates both. They do execute on every pull request, just not from ci-guards.yml:

test_ci_actionlint_covers_every_workflow.py runs in testbox-broker-guard.yml, which carries an explicit comment that it must run on every pull request with no path filter
test_ci_selective_layer_wiring.py runs in ci-artifact-transport.yml

The reason they cannot be linux-guard here is a validator limitation, not a fact about the tests: validate_test_execution_registry.py checks the linux-guard lane against CI_GUARDS.read_text() alone, so a test that demonstrably runs on every PR from another always-on workflow gets rejected. #13710 replaces that with the text of every workflow and rewords the error to "is not run by any workflow", which lets both keep an accurate lane. Routing them to legacy instead satisfies the same validator by relabelling a live guard as unmigrated.

#13710 is also the base of a stack: #13714 (201e8d41b14) sits directly on f283ff41213 from #13710, which is why #13714's workflow-guard-tests / preflight is green while pull requests based on current main are red.

Suggest closing this one and landing #13710, then #13714.

For context on urgency: the registry guard landed today at 12:34 in #13615, and nine tests have reached main since without an entry — from #13701, #13424, #13567, #13692 and others. python3 scripts/ci/validate_test_execution_registry.py exits 1 on a clean main checkout today. It is not that main itself runs this guard — ci-guards.yml is workflow_call only — but any pull request rebased onto current main fails guards / workflow-guard-tests / preflight (#13715 and #13697 do now). PRs whose last CI predates those merges still show green and will flip on the next run.

teamleaderleo · 2026-09-22T18:25:46Z

Closing as stale — every change here is already on main.

Verified against main at e6b3d6b394:

All nine tests this PR registers now have entries in tests/test-execution.toml. ci: register the nine tests main's execution registry does not know about #13710 registered the batch that ci: declare Python test execution lanes #13615 landed; ci: register the R2 cache census in the test execution registry #13731 and ci: register the settings JSONC and test-wiring regressions #13739 covered the later arrivals.
The step this PR adds to .github/workflows/ci-guards.yml — python3 tests/test_ios_screenshot_capture_guard.py under matrix.group == 'release-ios' — already exists on main at ci-guards.yml:456, and the test is registered as linux-guard. The "runs nowhere" gap this PR opened is closed.

So the diff would be a no-op plus a duplicate guard step.

What is still true and worth keeping. Two things from the analysis here outlive the patch:

validate_test_execution_registry.py still fails on main today, for a different reason: tests/test_sync_test_wiring.py is registered twice (lines 534 and 898). That is the same "one unregistered/malformed file turns every open PR red" failure mode this PR was opened to clear. ci: run the renderer-memory gate and unblock the test registry #13747 removes the duplicate; ci: fail only the pull request that adds an unregistered test #13745 removes the blast radius so a future instance only fails the PR that caused it.
The legacy lane is still an inventory bucket, not a lane — 61 entries on main. The specific observation from this PR stands: the tests invoked by testbox-broker-guard.yml and ci-artifact-transport.yml are filed as legacy because the registry models no lane for those workflows. Modelling real lanes for them is still unclaimed work, and ci: fail only the pull request that adds an unregistered test #13745 is the natural place for it since it now owns the validator.

I have carried both points onto #13745.

Nothing here is lost; reopen if I have misread the registry state.

teamleaderleo requested a review from lawrencecchen as a code owner September 22, 2026 17:00

This was referenced Sep 22, 2026

chore(tests): escape the shell backslash in the git stub heredoc #13720

Merged

ci: make the compile cache key survive a change of runner pool #13723

Merged

ci: register the two tests that make main fail its own guard #13735

Closed

teamleaderleo closed this Sep 22, 2026

Navigation Menu

Repository navigation

ci: register the nine unregistered Python tests - #13712

ci: register the nine unregistered Python tests#13712
teamleaderleo wants to merge 1 commit into
mainfrom
ci/register-unregistered-tests

teamleaderleo commented Sep 22, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

coderabbitai Bot commented Sep 22, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

teamleaderleo commented Sep 22, 2026

Uh oh!

teamleaderleo commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

teamleaderleo commented Sep 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How each was classified

Validation

Summary by cubic

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

coderabbitai Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

teamleaderleo commented Sep 22, 2026

Uh oh!

teamleaderleo commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Sep 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 22, 2026 •

edited

Loading