Skip to content

ci: reconcile fleet Linux pools from live probe receipts - #30714

Merged
teamleaderleo merged 17 commits into
feat-cmux-nextfrom
lane/runner-pools-health-fleet
Nov 1, 2026
Merged

teamleaderleo merged 17 commits into
feat-cmux-nextfrom
lane/runner-pools-health-fleet

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

A stalled label remains dead when probe health ages or an API read fails. The reconciler reads current repository state before writing, accepts a fleet start or recorded three-minute no-start, and keeps legacy secret-bearing jobs on a separate ephemeral pool. When only the fleet is available, it publishes that safe route and explicitly reports legacy capacity unavailable.

Leo: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT.”

The entry point runs from the trusted tooling checkout beside main's existing policy. The outage switch still evaluates Blacksmith alone; a fleet probe cannot count as Blacksmith recovery. The writer preserves promotion and unknown metadata and records bounded route changes for the existing queued-job rescuer. It changes no runner trust, secrets, workflow permissions or started-run cancellation policy. Main workflow/schema wiring lands separately before activation.

Validation: 20 targeted Python integration tests and 8 cloud-watch rescue tests pass against the current main policy and the feat runtime, covering actual-start/no-start evidence, stale health, unknown reads, partial capacity, mirror recovery, concurrent state changes and bounded move receipts. Independent Astra review: SHIP at efe47cc. Existing compile caches are unaffected; this performs no compile or build.

Performance Before After Evidence
Blacksmith minutes/day Not measured Not measured Routing policy only; no unsupported usage estimate
Minimum recorded fleet no-start before overflow No fleet probe 3 minutes Targeted probe-receipt tests

Agent session: agent session
🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Reconciles fleet Linux runner pools from live probe receipts so a stalled label no longer stays dead when probe health ages or an API read fails.

  • The reconciler reads current repository state before any write and accepts only a fleet start or a recorded three-minute no-start.
  • Legacy secret-bearing jobs stay on a separate ephemeral pool; when only the fleet is available, it publishes that route and explicitly reports legacy capacity unavailable.
  • Adds a bounded rescue for the cloud-overflow probe watcher that only touches an unstarted watcher job.

Written for commit 25d4aba. Summary will update on new commits.

View guided diff Turn on auto-fix

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, codex gpt-6-astra high): SHIP at efe47cc. The dependency blocker is fixed: main Python bootstrap exists, is sparse-checked out, and is invoked correctly. All 20 integration tests pass through it with no skips; all 8 scoped rescue tests pass. No remaining findings.

…-pools-health-fleet

# Conflicts:
#	tests/test-execution.toml
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, codex gpt-6-astra high): SHIP at 25d4aba. Base merge preserves both test-registration lists; runtime and workflow logic remain unchanged. All 28 focused tests pass, TOML registrations are unique, and the tree is clean. No findings.

@teamleaderleo
teamleaderleo merged commit 42d132c into feat-cmux-next Nov 1, 2026
56 of 71 checks passed
@teamleaderleo
teamleaderleo deleted the lane/runner-pools-health-fleet branch November 1, 2026 10:17
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 25d4aba: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30714 -f ref=lane/runner-pools-health-fleet -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base ecf95af; does not block

Shipped in:

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 25d4abaa Deployed Nov 1, 2026 by teamleaderleo via cmux-tui tree preflight #95601
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant