ci: read the gui backlog eight runs at a time in late placement

[teamleaderleo]

What

After admission, late placement counts the gui jobs already queued so it can move the shards that would wait more than one round to the retry runner. gui_backlog read up to 30 in-flight runs' job lists one request at a time. Under load that outran the step's timeout-minutes: 1:

• In the last 21 PR runs where the step ran, it took p50 55 s and p90 73 s. 10 of the 21 hit the one-minute cap.
• A timed-out step writes no runners output, so nothing moves. Every shard stays queued on the owned gui runners, one per runner. In run 36385188925 the seven shards started 8 to 30 minutes after admission, and shard 1 waited until the rescue re-run.

Reads now run eight at a time (BACKLOG_READERS), oldest batch first. They stop once the count reaches enough, the only value decide() compares against, so any extra counted past it changes nothing. Up to 30 reads now take about four request round trips instead of thirty.

Tests

• New test_backlog_reads_runs_concurrently_oldest_batch_first_and_stops_when_enough: 20 runs. A threading.Barrier makes all eight reads of a batch be in flight at once, and only the oldest batch is read.
• New test_a_failed_backlog_read_raises_so_nothing_moves: a reader thread's error still reaches decide(), which warns and moves nothing.
• The existing backlog test now expects the whole first batch to be read (4 runs, count 8 instead of stopping at 6).
• tests/test_ci_late_placement.py: 32 OK. The concurrency test and the updated backlog test fail on main. scripts/check-test-determinism.py is clean.

🤖 Generated with Claude Code

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 04ce9eea-7f16-4fdb-b872-f3fae664c49a

📥 Commits

Reviewing files that changed from the base of the PR and between 714ec53 and ca95bb7.

📒 Files selected for processing (2)

• scripts/ci/late_placement.py
• tests/test_ci_late_placement.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

---

📝 Walkthrough

Walkthrough

gui_backlog now reads run job lists concurrently in batches of up to eight, processing runs oldest first. It stops before the next batch when the accumulated count reaches the threshold. The final batch can cause the count to exceed the threshold.

Changes

GUI backlog reading

Layer / File(s)	Summary
Batch backlog reads scripts/ci/late_placement.py, tests/test_ci_late_placement.py	gui_backlog reads up to eight runs concurrently, oldest first, and counts queued jobs with the requested label. Tests check counted run IDs, concurrent reads, stopping between batches, and propagated read failures.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to ca95b

The bounded concurrent reads appear mergeable after normal checks; no actionable risk remains from the reviewed change.

Security Architecture Review

Security architecture risk: 🔵 Low · up to ca95b

The change keeps the existing runner-selection rules and access boundaries, while making backlog reads concurrent. It may improve placement timeliness, but the effect of eight simultaneous requests on API limits has not been measured.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The changed request pattern is bounded to eight concurrent reads among at most the configured backlog lookups. Its direct outcome is the CI runner-placement decision, not a new credential or write operation.

Trust Boundaries and Controls

• inferred — No new attacker-callable boundary is established by the changed constant or internal reader: job-list requests still use run IDs obtained from the existing workflow-run listing and the same repository-scoped authenticated client.

Resilience and Maintainability Implications

• observed — If a backlog read fails, placement does not act on a partial count. The existing no-overflow fallback remains in effect.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (24 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input	✅ Passed	PASS: The pull request changes only CI late-placement backlog reads and their tests. The diff introduces batched ThreadPoolExecutor reads for GitHub job lists; it does not change Cloud terminal crea…
Cmux Swift Actor Isolation	✅ Passed	PASS: The review-scoped diff changes only Python CI code and its Python tests. It contains no Swift files or production Swift changes, so it cannot introduce or worsen the specified Swift actor-isolat…
Cmux Swift Blocking Runtime	✅ Passed	PASS: The pull request changes only Python files (scripts/ci/late_placement.py and tests/test_ci_late_placement.py). It introduces no production Swift changes, so the Swift blocking-runtime check …
Cmux Browser Automation Off-Main	✅ Passed	The PR changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The diff contains no browser socket commands, WebKit/AppKit access, main-actor routing, worker-router changes,…
Cmux Expensive Synchronous Load	✅ Passed	The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. It contains no production Swift changes, so the expensive synchronous Swift load check does not appl…
Cmux Cache Substitution Correctness	✅ Passed	The pull request changes only Python files: scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The custom check applies only to production Swift, TypeScript, and JavaScript changes,…
Cmux No Hacky Sleeps	✅ Passed	PASS. The production diff adds batched ThreadPoolExecutor reads and does not add sleeps, timers, polling, fixed backoff, or wall-clock synchronization. The only explicit timeout is `threading.Barrie…
Cmux Algorithmic Complexity	✅ Passed	The changed production path reads each selected run once in batches of up to BACKLOG_READERS (8). It scans each returned job list once and uses the explicit BACKLOG_LOOKUPS = 30 bound. The existin…
Cmux Swift Concurrency	✅ Passed	PASS: The authoritative PR diff changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. It contains no Swift files or Swift code. The new concurrency uses Python `concurrent…
Cmux Swift @Concurrent	✅ Passed	PASS: The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The diff contains no Swift files or Swift concurrency declarations, so the @concurrent check…
Cmux Swift Package Boundaries	✅ Passed	PASS: The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. It introduces no production Swift changes, so the Swift package boundary rule does not apply.
Cmux Swiftpm Lockfiles	✅ Passed	The PR changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. It does not change Package.swift, Package.resolved, .gitignore, Xcode project files, workflows, or depen…
Cmux Swift Logging	✅ Passed	PASS: The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py, both Python files. It adds no production Swift code or logging statements. The existing `print…
Cmux User-Facing Error Privacy	✅ Passed	PASS. The production diff changes internal CI backlog reads and adds concurrency; it does not add or alter cmux app, CLI, or product API error text. The only production warning remains the existing la…
Cmux Full Internationalization	✅ Passed	PASS: The authoritative diff changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The production changes implement concurrent CI backlog reads and update internal docstri…
Cmux Swiftui State Layout	✅ Passed	PASS: The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. Both are Python files. The diff contains no SwiftUI, Swift state, layout, or render-time mutat…
Cmux Architecture Rethink	✅ Passed	PASS: The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The authoritative diff contains no Swift files, so the Swift architectural-rethink failure con…
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	The PR changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The authoritative diff contains no Swift changes and no auxiliary window code, so the close-shortcut rule is n…
Cmux Source Artifacts	✅ Passed	The PR changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The diff contains hand-written source and test changes for batched backlog reads. It adds no logs, caches, bui…
Cmux No Test Or Debug Seam In Production Source	✅ Passed	The pull request changes only scripts/ci/late_placement.py and tests/test_ci_late_placement.py. The authoritative diff contains no Swift file under a production Sources/ path, so this check is n…
Title check	✅ Passed	The title clearly and concisely describes the main change: reading the GUI backlog eight runs at a time during late placement.
Description check	✅ Passed	The description explains the problem, resulting behavior, implementation approach, performance impact, and test coverage. It omits the template headings for Changelog, Demo Video, and Checklist, but t…

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

Dogfood build of ca95bb7ec20f4cf78beeee82508f5bae6491339e

cmux DEV pr-15207-ca95bb7e.app

The link opens this exact commit in the cmux dev menu bar app. The build starts on each push and the page waits until it is ready; a newer push replaces it. It signs in against production, so Cloud or backend changes still need a tagged build with a development backend.

[github-actions]

CI failure attribution

CI passes on ca95bb7ec2 (run 36392522088 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's.

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[github-actions]

Merge receipt for ca95bb7ec2: every check was green at merge (11 verified; 12 skipped by policy). Full suite runs on main after merge.