fix(ci): janitor cancels orphaned CI runs whose run-name says release= by austinywang · Pull Request #25250 · manaflow-ai/cmux

austinywang · 2026-10-26T02:09:19Z

Summary

Every ci.yml run on main since 10/23 sat pending with zero jobs. Run 40039647633 (a 10/23 main dispatch) lost its macos / macOS status job's Blacksmith assignment and stayed queued. It held the CI-refs/heads/main concurrency group, so each later main dispatch waited behind it until the next dispatch replaced it. That held up gh-merge-green's base verdict.

The CI queue janitor already cancels orphans like this, but it read the run as a release. orphan_protected_reason matched release|publish|notar against the run name and the path, and ci.yml's run-name contains release=arm64. The janitor's sweep listed it as keep (release/publish workflow; left for a human). RESERVED_POOL_WORKFLOW had the same bug and was already fixed to match only the path. This PR does the same for orphan protection.

Every active workflow whose name contains release, publish or notarization also has the word in its file path, so those stay protected.

The stuck run was cancelled by hand. The pending main run then started.

Testing

python3 -m unittest tests.test_ci_queue_janitor -k main_schedule_nightly_and_testflight -k release_merge_queue_and_tag

base (test commit deb90a7): fails, ['skip'] != ['cancel'] for the ci.yml main dispatch case
head: passes. The full file passes too (186 tests).

Observed / Not run

Ran: the focused test (red on base, green on head), the full tests/test_ci_queue_janitor.py, and scripts/verify-local.py (16 of 17 checks passed; feature-flags was interrupted at its 60 s timeout, and this change doesn't touch it).
Not run: a live janitor sweep with this code.

Changelog

none

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by CodeRabbit

Bug Fixes
- Orphaned CI runs are now protected based on their workflow path rather than terms in their run names. Runs with protected-looking names but unrelated workflow paths can now be cancelled.
- Added coverage confirming that orphaned nightly and TestFlight-related runs, including a workflow_dispatch run with a release label, are cancelled as expected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…low path ci.yml's run-name carries routing metadata (release=arm64), so the orphan pass kept every orphaned CI run as a release left for a human. Run 40039647633 on main held CI's concurrency group for two days, leaving every later main run pending with zero jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-26T02:14:10Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: manaflow-ai/cmux
Branch: main
File: .coderabbit.yaml
Commit: b70ae22
Layers: Repository YAML
Review profile: ASSERTIVE
Plan: Advanced
Run ID: ba409382-8a64-4112-bc39-2fe683460eb8

Commits

Reviewing files that changed from the base of the PR and between b70ae22 and 3743bb1.

Files selected for processing: 2

scripts/ci/queue_janitor.py
tests/test_ci_queue_janitor.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

The janitor now checks protected-workflow patterns against workflow paths only. A test covers cancellation of an orphan ci.yml run whose name includes release=arm64.

Changes: Orphan workflow protection

Layer / File(s)	Summary
Workflow path matching and validation `scripts/ci/queue_janitor.py`, `tests/test_ci_queue_janitor.py`	`orphan_protected_reason` now checks the workflow path alone. The test verifies that an orphan run with `release=arm64` in its name is cancelled.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Possibly related issues

manaflow-ai/cmux#14196: The issue tracks CI queue backup, and this change addresses an orphan classification that could hold the main concurrency group and delay later runs.

Suggested labels: risk:high

Suggested reviewers: teamleaderleo, lawrencecchen

Merge Risk: Minimal · up to 3743b

The workflow-path change is covered by the supplied regression test, and the lint concern is unsupported by repository configuration.

Pre-merge checks: 24 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 24 passed

Check name	Status	Explanation
Title check		The title names the main change: the janitor now cancels orphaned ci.yml runs whose run name contains release=, instead of keeping them as release workflows. It is 70 characters and 10 words, within t…
Description check		The description follows the template's main sections: Summary, Testing, and Changelog. Summary states the problem (stuck main runs from a release-misread orphan), the mechanism (orphan_protected_reaso…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input		PASS — The pull request changes only CI orphan detection and its regression test (`scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`). It does not change Cloud terminal creation, persi…
Cmux Swift Actor Isolation		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The production change is Python, and the added regression coverage is also Python. The diff cont…
Cmux Swift Blocking Runtime		PASS: The PR changes only Python files (`scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`). It introduces no Swift production change and therefore no blocking or timing-based Swift ru…
Cmux Browser Automation Off-Main		The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The diff contains no `browser.*` socket commands, WebKit/AppKit access, worker-router changes, or brow…
Cmux Expensive Synchronous Load		The pull request changes only Python files: `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. It adds no Swift code and no synchronous agent-history load on the main actor or an inte…
Cmux Cache Substitution Correctness		PASS. The pull request changes only Python (`scripts/ci/queue_janitor.py`) and Python tests. It does not change production Swift, TypeScript, or JavaScript, so the cache-substitution condition is inap…
Cmux No Hacky Sleeps		The diff introduces no sleep, timer, polling, fixed delay, or wall-clock wait. The production change only switches `orphan_protected_reason` to inspect `run.path`. The added test uses deterministic da…
Cmux Algorithmic Complexity		The diff does not introduce an algorithmic-complexity violation. It replaces a two-field string construction with one regex search over the workflow path in `scripts/ci/queue_janitor.py:2058`. The cha…
Cmux Swift Concurrency		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The diff contains no Swift code, so it introduces no legacy Swift concurrency pattern covered by…
Cmux Swift `@Concurrent`		The pull request changes only Python files: `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The diff contains no Swift files or Swift async isolation changes, so the `@concurrent` …
Cmux Swift Package Boundaries		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The diff contains no Swift production changes, so the Swift package-boundaries rule does not app…
Cmux Swiftpm Lockfiles		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. It does not change a SwiftPM package, Xcode project, `.gitignore`, workflow, or dependency file.…
Cmux Swift Logging		The pull request changes only Python files: `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. It adds no production Swift code and adds or changes no Swift logging.
Cmux User-Facing Error Privacy		PASS: The diff changes orphan cancellation policy in `scripts/ci/queue_janitor.py` and adds a regression test. It does not add or materially change a cmux user-facing error, alert, API body, or recove…
Cmux Full Internationalization		PASS. The pull request changes only `scripts/ci/queue_janitor.py` and its test. The production change updates an internal workflow-path matching condition and adds developer comments; it does not add …
Cmux Swiftui State Layout		PASS. The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The diff contains no SwiftUI, Swift, or state-layout changes, so the custom SwiftUI check is not…
Cmux Architecture Rethink		PASS: The pull request changes only Python CI janitor code and its Python tests. The reviewed diff contains no Swift files or Swift architectural changes, so the Swift architectural rethink failure co…
Cmux Swift Auxiliary Window Close Shortcuts		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. The reviewed diff contains no Swift or window implementation changes, so the auxiliary-window cl…
Cmux Source Artifacts		The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. These are intentional hand-written source and regression-test files. The diff adds no local output, ge…
Cmux No Test Or Debug Seam In Production Source		PASS: The pull request changes only `scripts/ci/queue_janitor.py` and `tests/test_ci_queue_janitor.py`. It changes no Swift file under a production `Sources/` path, so this check is not applicable.

Full details: Title check

Explanation

The title names the main change: the janitor now cancels orphaned ci.yml runs whose run name contains release=, instead of keeping them as release workflows. It is 70 characters and 10 words, within the concise range, and it uses a conventional fix(ci) prefix. It does not mention the path-only protection change, but that is a detail of the same fix.

Full details: Description check

Explanation

The description follows the template's main sections: Summary, Testing, and Changelog. Summary states the problem (stuck main runs from a release-misread orphan), the mechanism (orphan_protected_reason matching the run name), and the resulting behavior. Testing separates the focused red-on-base and green-on-head test, the full file run, and an Observed / Not run section, as the template requires for bug fixes. Changelog is set to none, which the template allows for internal CI changes. The Proof section and most checklist items are omitted without a stated reason. The Proof section needs a one-line reason, such as 'CI only', because the template asks for one. The checklist is not addressed, though most items may not apply to a CI-only change.

Full details: Cmux Cloud Persistent Session And Early Input

Explanation

PASS — The pull request changes only CI orphan detection and its regression test (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It does not change Cloud terminal creation, persistent cmux-tui transport, manual renderer admission, input ownership, or auth/idempotency/revision/lease handling.

Full details: Cmux Swift Actor Isolation

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The production change is Python, and the added regression coverage is also Python. The diff contains no Swift changes, so it does not introduce or worsen Swift 6 actor isolation issues.

Full details: Cmux Swift Blocking Runtime

Explanation

PASS: The PR changes only Python files (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It introduces no Swift production change and therefore no blocking or timing-based Swift runtime synchronization.

Full details: Cmux Browser Automation Off-Main

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no browser.* socket commands, WebKit/AppKit access, worker-router changes, or browser policy tests. The custom check is therefore not applicable.

Full details: Cmux Expensive Synchronous Load

Explanation

The pull request changes only Python files: scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It adds no Swift code and no synchronous agent-history load on the main actor or an interactive path.

Full details: Cmux Cache Substitution Correctness

Explanation

PASS. The pull request changes only Python (scripts/ci/queue_janitor.py) and Python tests. It does not change production Swift, TypeScript, or JavaScript, so the cache-substitution condition is inapplicable.

Full details: Cmux No Hacky Sleeps

Explanation

The diff introduces no sleep, timer, polling, fixed delay, or wall-clock wait. The production change only switches orphan_protected_reason to inspect run.path. The added test uses deterministic data. An existing cancel_orphans sleep remains unchanged from the base revision.

Full details: Cmux Algorithmic Complexity

Explanation

The diff does not introduce an algorithmic-complexity violation. It replaces a two-field string construction with one regex search over the workflow path in scripts/ci/queue_janitor.py:2058. The change performs no nested collection scan, repeated sort/filter, batch rescan, or in-memory join. The added test is test-only scaffolding.

Full details: Cmux Swift Concurrency

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift code, so it introduces no legacy Swift concurrency pattern covered by this check.

Full details: Cmux Swift `@Concurrent`

Explanation

The pull request changes only Python files: scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift files or Swift async isolation changes, so the @concurrent check is not applicable.

Full details: Cmux Swift Package Boundaries

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift production changes, so the Swift package-boundaries rule does not apply.

Full details: Cmux Swiftpm Lockfiles

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It does not change a SwiftPM package, Xcode project, .gitignore, workflow, or dependency file. The SwiftPM lockfile conditions are therefore not applicable.

Full details: Cmux User-Facing Error Privacy

Explanation

PASS: The diff changes orphan cancellation policy in scripts/ci/queue_janitor.py and adds a regression test. It does not add or materially change a cmux user-facing error, alert, API body, or recovery message. The janitor writes diagnostics to the internal GitHub Actions step summary, which is an operator-only CI surface. The added comments and test fixture are also developer-only.

Full details: Cmux Full Internationalization

Explanation

PASS. The pull request changes only scripts/ci/queue_janitor.py and its test. The production change updates an internal workflow-path matching condition and adds developer comments; it does not add or change user-facing text, localization catalogs, web messages, API copy, markdown, or changelog content. The test-only additions are explicitly allowed.

Full details: Cmux Swiftui State Layout

Explanation

PASS. The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no SwiftUI, Swift, or state-layout changes, so the custom SwiftUI check is not applicable.

Full details: Cmux Architecture Rethink

Explanation

PASS: The pull request changes only Python CI janitor code and its Python tests. The reviewed diff contains no Swift files or Swift architectural changes, so the Swift architectural rethink failure conditions do not apply.

Full details: Cmux Swift Auxiliary Window Close Shortcuts

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The reviewed diff contains no Swift or window implementation changes, so the auxiliary-window close-shortcut rule does not apply.

Full details: Cmux Source Artifacts

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. These are intentional hand-written source and regression-test files. The diff adds no local output, generated artifacts, scratch directories, caches, build output, or copied files.

Finishing Touches: 💡 2

📝 Generate docstrings: 💡

Commit to this branch
Create a new PR

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-26T05:27:52Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

austinywang · 2026-10-26T19:18:46Z

Review (independent subagent, correctness first) on head 3743bb1: SHIP.

orphan_protected_reason now matches only the workflow path. Every real release, publish or notarization workflow has the word in its file name, so each one stays protected. That covers reusable workflows too, because they run under the caller's path.
Besides ci.yml, one more workflow loses protection: cmux-next-dev-daily.yml, whose run-name says "publish". A cancelled orphan of it is safe: cmux-next-dev-daily-dispatch.yml re-dispatches the publisher on its next tick. Cancelling also unblocks the dispatcher, which counts a queued run as already sent. Accepted as is.
Not blocking, follow-up: the general protected_reason still matches name plus path, so ci.yml runs read as release workflows to the non-orphan policy. That errs toward protecting too much, which is safe.
Tests: python3 -m unittest tests.test_ci_queue_janitor passes all 186 tests.

github-actions · 2026-10-26T19:28:22Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Merge receipt for 3743bb131f: every check was green at merge (9 verified; 17 skipped by policy). Full suite runs on main after merge.

austinywang and others added 2 commits October 25, 2026 19:03

test(ci): orphaned main CI dispatch must not read as a release

deb90a7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

austinywang merged commit f891e52 into main Oct 26, 2026
50 of 54 checks passed

austinywang deleted the janitor-orphan-run-name branch October 26, 2026 19:28

coderabbitai Bot mentioned this pull request Oct 27, 2026

ci: never cancel a nightly, DEV or promote run as a queue-janitor orphan #25882

Merged

Navigation Menu

Repository navigation

fix(ci): janitor cancels orphaned CI runs whose run-name says release= - #25250

fix(ci): janitor cancels orphaned CI runs whose run-name says release=#25250
austinywang merged 2 commits into
mainfrom
janitor-orphan-run-name

austinywang commented Oct 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

austinywang commented Oct 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

austinywang commented Oct 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Observed / Not run

Changelog

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

austinywang commented Oct 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

austinywang commented Oct 26, 2026 •

edited by coderabbitai Bot

Loading