Skip to content

fix(ci): refresh runner capacity after nightly admission waits - #31891

Merged
teamleaderleo merged 4 commits into
feat-cmux-nextfrom
lane/runner-state-nightly-rechecks
Nov 2, 2026
Merged

teamleaderleo merged 4 commits into
feat-cmux-nextfrom
lane/runner-state-nightly-rechecks

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Portable placement, nightly control admission and the dedicated nightly build wait can reuse a workflow's fixed runner document after sleeping. A busy-to-idle transition is then invisible until overflow or timeout.

Keep the shared document for the first look and request live runner capacity after each existing wait. Injected test readers keep their existing call contract. This extends the first-look policy in #31889 and depends on its shared client option; this PR is stacked on that branch.

Performance

metric before after delta harness or run link
routing-App runner-list requests/h at 1000 idle first looks/h with fresh state 0 0 0 same document-backed picker entrypoint fixtures
routing-App requests/h at 1000 busy control waits/h, capacity idle on first recheck 0 3000 +3000, intentional live recheck test_busy_document_is_rechecked_live_after_waiting, one visible group and one-page lists
portable/control placement when capacity becomes idle at first recheck overflow after 180 s fleet at 30 s restores fleet placement busy-document/live-idle entrypoint regressions
dedicated build placement when capacity becomes idle at first recheck no dedicated placement after 60 s dedicated placement at 30 s restores dedicated placement busy-document/live-idle build regression

The live recheck trades requests for correctness only after capacity was unavailable. No new timer or wait is introduced; existing waits retain their deadlines and cleanup. This table models the named fixture rate, not total installation traffic. No compilation or cache behavior changes.

Observed / Not run

All 38 portable placement, 11 nightly control and 59 nightly build tests pass. The three new busy-document regressions fail at test-only commit 8ec0edd and pass with the fix. git diff --check passes. No application build is needed for these picker reads.

Changelog

none

Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes nightly admission waits reusing a workflow's fixed runner document after sleeping, so a busy-to-idle transition no longer stays invisible until overflow or timeout.

Portable placement, nightly control, and the dedicated nightly build picker now keep the shared document only for their first look, then request live runner capacity on each recheck. Runners that become idle during the existing wait window are now found at the first recheck instead of after overflow or timeout, and no new timers or waits are introduced. Injected test readers keep their existing call contract.

Written for commit a504c64. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 4 commits November 2, 2026 05:56
Use shared repository state for the archive picker first look and preserve repository-only scope. DEV placement rechecks bypass the fixed workflow document after waiting.

Co-Authored-By: OpenAI GPT-6.1 Sol <noreply@openai.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-44e895a13a32951aa9ae
Retain the shared-state first look, then bypass the fixed workflow observation when portable, control and dedicated build pickers recheck busy capacity.

Co-Authored-By: OpenAI GPT-6.1 Sol <noreply@openai.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-44e895a13a32951aa9ae
Base automatically changed from lane/runner-state-live-rechecks to feat-cmux-next November 2, 2026 06:11
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at a504c64

Checked: in portable_job_placement.main, nightly_next_control_runner.choose and the nightly_build_runner dedicated wait, the first look uses the document and every look after a sleep reads live. The idle path makes zero calls, and a missing or stale document behaves as before. Only scripts and tests change. Tests pass (portable 38, nightly_build_runner 59, nightly_next_control_runner 11), and the new ones fail at test-only 8ec0edd.

Follow-ups:

  • [follow-up] nightly_build_runner.main: the first-look flag is shared with the later owned_route read, which goes live even when no wait happened. That costs a few API calls; the rule should be 'use the document until a sleep has happened'.
  • [follow-up] The nightly_build_runner test should assert the first look made zero org runner-list calls.
  • [follow-up] fast_checks_runner has the same fixed-document wait.

@teamleaderleo
teamleaderleo merged commit 1580137 into feat-cmux-next Nov 2, 2026
75 checks passed
@teamleaderleo
teamleaderleo deleted the lane/runner-state-nightly-rechecks branch November 2, 2026 06:11
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Shipped in:

This branch was previously deployed

1 inactive deployment
ci-routing-agent — a504c64d Deployed Nov 2, 2026 by teamleaderleo via wait for same-key tree publisher #99641
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant