[teamleaderleo]

What

The queue janitor's macos-pool-load snapshot counts a queued macOS job as reserved_queued only when its run's workflow is freshness work. The cmux-next placement (scripts/ci/next_macos_cascade.py) keeps PR overflow off a Blacksmith macOS label while that count is above 0. Until now the reserved set was a regex matching only release.yml and nightly.yml, so .github/workflows/auto-resume-nightly-notarization.yml ("Continue nightly notarization") didn't count. On 2026-10-30 at 12:43Z its three wait-and-staple jobs queued on blacksmith-6vcpu-macos-26 behind 22 PR overflow jobs, and nothing kept the label free for them.

Change

• scripts/ci/queue_janitor.py: the regex RESERVED_POOL_WORKFLOW is replaced by reserves_pool(path). It returns true for release.yml (the declared RESERVED_RELEASE_WORKFLOWS) and for every delivery workflow in scripts/ci/delivery_workflows.py's existing DELIVERY_GLOBS. That file is already the one list the janitor uses to decide which runs it must never cancel. Matching still uses the workflow path only, so a CI run whose name contains release=arm64 still doesn't reserve a label.
• Workflows that now reserve and didn't before, each with a Blacksmith macOS job: auto-resume-nightly-notarization.yml, resume-nightly-notarization.yml, nightly-next-canary.yml, nightly-next-launch-smoke.yml, nightly-next-train.yml and cmux-next-dev-daily.yml (DEV verify and tip DMG/deltas; the cascade already counts DEV's Release compile as freshness work in FRESH_JOBS).
• Not reserved, on purpose: the iOS TestFlight and App Store workflows (they're iOS publishing, not a macOS feed's freshness), and the DEV source build of cmux-next.yml (matching is by path only, and that path is the PR workflow).

Tests

New ReservedPoolWorkflowTests in tests/test_ci_queue_janitor.py:

• A queued auto-resume-nightly-notarization.yml job on blacksmith-6vcpu-macos-26 counts as reserved_queued (the same holds for resume-nightly-notarization.yml).
• An invariant over the real .github/workflows: every workflow whose file name or name: marks it as nightly, release, promote or notarization work, and that has a job whose runs-on can resolve to a Blacksmith macOS label (a literal blacksmith-*macos* or a MACOS_RUNNER* variable), must reserve its label. The only exceptions are listed with reasons: iroh-release-gate.yml, a manual test gate, and tmux-corpus.yml, a manual corpus test titled "terminal nightly". A new nightly or notarization workflow can't be missed.
• The PR workflows ci.yml, cmux-next.yml and ci-macos.yml don't reserve (a run name containing release=arm64 is used).

Evidence, run locally:

• Base 4ae385f8 with the new tests: 7 failures (0 != 1 for auto-resume, resume, nightly-next canary, launch smoke and train).
• Head: tests/test_ci_queue_janitor.py 251 OK, tests/test_ci_pr_runner_pool.py 252 OK, tests/test_ci_simple_pool_picker.py 46 OK, tests/test_delivery_workflows_cancel.py 25 OK, tests/test_nightly_delivery_concurrency.py PASS.

feat-cmux-next

feat-cmux-next has an older copy of queue_janitor.py with the same regex. It doesn't need this change: the janitor runs only on schedule, workflow_run (CI requested) and workflow_dispatch, and the first two always run from the default branch. The snapshot every branch's placement reads is built by main's copy. The feat copy will pick this up the next time main is merged into it.

Risk: low. When a nightly, DEV or notarization job is queued on a Blacksmith macOS label, PR overflow now waits instead of taking that label. This is what the cascade already does for nightly.yml and release.yml.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes the queue janitor's macos-pool-load snapshot so nightly notarization follow-ups and other delivery workflows reserve their Blacksmith macOS labels. Previously only release.yml and nightly.yml reserved; on 2026-10-30 auto-resume-nightly-notarization.yml's wait-and-staple jobs queued behind 22 PR overflow jobs on blacksmith-6vcpu-macos-26 because their workflow wasn't in the reserved set.

• Replaces the RESERVED_POOL_WORKFLOW regex with reserves_pool(), which returns true for release.yml and every workflow in delivery_workflows.py's DELIVERY_GLOBS (nightly, its verify/notarization follow-ups, DEV, and promote). Workflows are still matched by path only.
• Adds ReservedPoolWorkflowTests that fail on any workflow named or filed as nightly, release, promote, or notarization with a Blacksmith macOS job that isn't either reserved or explicitly exempted (e.g., iroh-release-gate.yml, tmux-corpus.yml).

^{Written for commit 6fb67ae. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • CI capacity tracking now accounts for queued jobs across release, nightly, development, promotion, and notarization workflows, while continuing to exclude designated workflows. This gives a more accurate view of available build capacity and helps prevent queued work from being overlooked when the system tracks resource availability.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Owned by an agent session · session

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: manaflow-ai/cmux
• Branch: lane/janitor-reserve-notarization
• File: .coderabbit.yaml
• Commit: 6fb67ae
• Layers: Repository YAML
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: f538ae13-9a33-488b-a30d-b263ea932974

Commits

Reviewing files that changed from the base of the PR and between 4ae385f and 6fb67ae.

Files selected for processing: 2

• scripts/ci/queue_janitor.py
• tests/test_ci_queue_janitor.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

Walkthrough

The queue janitor now counts queued jobs from release and delivery workflows as reserved. Tests cover nightly notarization workflows, configured exemptions, and listed pull-request workflows.

Changes: Workflow reservation classification

Layer / File(s)	Summary
Classify and verify reserved workflows scripts/ci/queue_janitor.py, tests/test_ci_queue_janitor.py	The janitor uses reserves_pool() to classify workflow paths when building pool snapshots. Tests check reservation counts for marked workflows, exemptions, and pull-request workflows.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related issues

• Issue 26576 concerns nightly notarization jobs queued on the same Blacksmith macOS label. This change updates how the janitor counts those queued jobs, but does not address the issue’s sweeper adoption or routing behavior.

Suggested reviewers: austinywang, lawrencecchen

Merge Risk: Minimal · up to 6fb67

The change reserves capacity for filename-matched delivery workflows while preserving PR-workflow exclusions. No actionable merge-blocking issue remains; merge after normal checks.

Pre-merge checks: 24 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 24 passed

Check name	Status	Explanation
Title check		The title clearly identifies the main change: nightly notarization follow-ups now reserve their Blacksmith macOS label through the CI janitor.
Description check		The description clearly explains the problem, implementation, affected workflows, exclusions, risk, and test results. It does not use the template headings for Changelog, Proof, or Checklist, but the …
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input		PASS. The pull request changes only CI queue reservation logic and its tests in scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff does not change Cloud terminal creation, cm…
Cmux Swift Actor Isolation		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It introduces no production Swift changes, so it cannot introduce or worsen Swift 6 actor-isolation mi…
Cmux Swift Blocking Runtime		PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift files or Swift runtime synchronization changes. The blocking-runtime …
Cmux Browser Automation Off-Main		PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no browser.* commands, socket-worker routing, WebKit/AppKit access, or brows…
Cmux Expensive Synchronous Load		PASS: The pull request changes only Python files (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It adds no production Swift code or agent-history load onto a main-actor or inter…
Cmux Cache Substitution Correctness		PASS: The pull request changes only Python files (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It does not contain production Swift, TypeScript, or JavaScript changes, so the c…
Cmux No Hacky Sleeps		The production diff changes workflow-path matching and reserved queue accounting only. It adds no sleep, timer, polling, fixed delay, or wall-clock wait. The timing-related RETRY_DELAYS code is exis…
Cmux Algorithmic Complexity		The diff does not introduce a prohibited scalable-collection algorithm. reserves_pool() performs path normalization and is_delivery_workflow() checks four fixed DELIVERY_GLOBS patterns. The call…
Cmux Swift Concurrency		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. No Swift files or Swift concurrency code changed, so the cmux Swift concurrency check is not applicabl…
Cmux Swift @Concurrent		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The reviewed diff contains no Swift files, so the @concurrent check is not applicable.
Cmux Swift Package Boundaries		The pull request changes only Python CI code and Python tests: scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift production changes, so it does not introdu…
Cmux Swiftpm Lockfiles		PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It does not change a SwiftPM package, .gitignore, Xcode project package references, dependency…
Cmux Swift Logging		The authoritative PR diff changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. These are Python files, and the diff adds no Swift logging or runtime diagnostic output. Ther…
Cmux User-Facing Error Privacy		PASS: The production diff changes internal CI queue accounting only. It adds reserves_pool() and updates reserved_queued in the macOS pool snapshot; it does not add or modify a cmux app UI, produc…
Cmux Full Internationalization		The diff changes only scripts/ci/queue_janitor.py and its tests. It adds CI workflow-path logic, operational comments, and test assertions. It does not add or change Swift UI text, string catalogs, …
Cmux Swiftui State Layout		PASS: The PR changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The reviewed diff contains no Swift or SwiftUI changes, so the SwiftUI state-layout rules do not apply.
Cmux Architecture Rethink		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It introduces no Swift architecture changes, so the Swift architectural rethink failure conditions do …
Cmux Swift Auxiliary Window Close Shortcuts		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The reviewed diff contains no Swift or SwiftUI window changes, so the auxiliary-window close-shortcut …
Cmux Source Artifacts		The diff changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. These are hand-written source and test files. No logs, screenshots, recordings, temporary directories, caches,…
Cmux No Test Or Debug Seam In Production Source		The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It changes no Swift file under a production Sources/ path, so the specified production test/debug se…

Full details: Description check

Explanation

The description clearly explains the problem, implementation, affected workflows, exclusions, risk, and test results. It does not use the template headings for Changelog, Proof, or Checklist, but the core required information is present and the change is well documented.

Full details: Cmux Cloud Persistent Session And Early Input

Explanation

PASS. The pull request changes only CI queue reservation logic and its tests in scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff does not change Cloud terminal creation, cmux-tui transport, manual renderers, input routing, or session fences.

Full details: Cmux Swift Actor Isolation

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It introduces no production Swift changes, so it cannot introduce or worsen Swift 6 actor-isolation mistakes.

Full details: Cmux Swift Blocking Runtime

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift files or Swift runtime synchronization changes. The blocking-runtime failure condition is not applicable.

Full details: Cmux Browser Automation Off-Main

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no browser.* commands, socket-worker routing, WebKit/AppKit access, or browser automation policy changes. The WebKit wait rule is therefore not applicable.

Full details: Cmux Expensive Synchronous Load

Explanation

PASS: The pull request changes only Python files (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It adds no production Swift code or agent-history load onto a main-actor or interactive path.

Full details: Cmux Cache Substitution Correctness

Explanation

PASS: The pull request changes only Python files (scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py). It does not contain production Swift, TypeScript, or JavaScript changes, so the cache-substitution correctness condition does not apply.

Full details: Cmux No Hacky Sleeps

Explanation

The production diff changes workflow-path matching and reserved queue accounting only. It adds no sleep, timer, polling, fixed delay, or wall-clock wait. The timing-related RETRY_DELAYS code is existing and unchanged. The added test data uses timestamps but does not add runtime synchronization.

Full details: Cmux Algorithmic Complexity

Explanation

The diff does not introduce a prohibited scalable-collection algorithm. reserves_pool() performs path normalization and is_delivery_workflow() checks four fixed DELIVERY_GLOBS patterns. The call occurs once per existing run in pool_load_snapshot() and does not rescan jobs or other user-owned records. The added workflow scan is test-only.

Full details: Cmux Swift Concurrency

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. No Swift files or Swift concurrency code changed, so the cmux Swift concurrency check is not applicable.

Full details: Cmux Swift Package Boundaries

Explanation

The pull request changes only Python CI code and Python tests: scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The diff contains no Swift production changes, so it does not introduce or expand any Swift package-boundary violation.

Full details: Cmux Swiftpm Lockfiles

Explanation

PASS: The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It does not change a SwiftPM package, .gitignore, Xcode project package references, dependency pins, or a workflow file. The SwiftPM lockfile policy has no applicable failure condition.

Full details: Cmux Swift Logging

Explanation

The authoritative PR diff changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. These are Python files, and the diff adds no Swift logging or runtime diagnostic output. Therefore .github/review-bot-rules/swift-logging.md is not applicable.

Full details: Cmux User-Facing Error Privacy

Explanation

PASS: The production diff changes internal CI queue accounting only. It adds reserves_pool() and updates reserved_queued in the macOS pool snapshot; it does not add or modify a cmux app UI, product CLI, API error, alert, command output, or recovery message. The added Blacksmith and workflow details are in CI comments and tests, which the rule allows. No concrete path to a cmux end user exists.

Full details: Cmux Full Internationalization

Explanation

The diff changes only scripts/ci/queue_janitor.py and its tests. It adds CI workflow-path logic, operational comments, and test assertions. It does not add or change Swift UI text, string catalogs, web UI/API copy, metadata, rendered markdown, changelog text, or locale files. The internationalization check therefore does not apply.

Full details: Cmux Architecture Rethink

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It introduces no Swift architecture changes, so the Swift architectural rethink failure conditions do not apply.

Full details: Cmux Swift Auxiliary Window Close Shortcuts

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. The reviewed diff contains no Swift or SwiftUI window changes, so the auxiliary-window close-shortcut rule does not apply.

Full details: Cmux Source Artifacts

Explanation

The diff changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. These are hand-written source and test files. No logs, screenshots, recordings, temporary directories, caches, build output, dependency checkouts, or other source-control artifacts enter the diff.

Full details: Cmux No Test Or Debug Seam In Production Source

Explanation

The pull request changes only scripts/ci/queue_janitor.py and tests/test_ci_queue_janitor.py. It changes no Swift file under a production Sources/ path, so the specified production test/debug seam condition is not applicable.

Full details: Docstring Coverage

Explanation

Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}