Skip to content

ci: route nightly preparation through fleet placement - #30113

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/nightly-prepare-attest-routing
Nov 1, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/nightly-prepare-attest-routing

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

NEXT nightly preparation currently resolves to fixed Blacksmith Linux labels. That leaves portable, secret-free preparation work waiting on paid capacity even though the nightly placement step already selects free owned minis.

This routes prepare-nightly-variant-primary and prepare-nightly-variant through the existing bounded nightly placement. They use an idle owned mini when selected and retain the measured Blacksmith Linux expression as overflow. The placement waits up to five minutes only while runner inventory and the janitor snapshot are fresh; unknown or stale capacity falls through immediately to Blacksmith. Attestation remains on its trusted Blacksmith route because it needs id-token and attestations: write. The routing table and debt guard classify both preparation jobs as portable checks. The current REQUIRE_LIVE_CAPACITY contract from #30173 is preserved for preparation overflow and the existing admission fallback remains unchanged.

Leo said: “PUBLISH THE F***ING NIGHTLY” and asked where the efficiency improvements were. This removes two avoidable Blacksmith-only waits without changing secrets, permissions, signing routes, or promotion behavior.

Validation: tests/test_nightly_linux_admission.py (25), tests/test_runner_routing_table.py (27), tests/test_ci_portable_job_placement.py (30), tests/test_nightly_trusted_routing.py (33), and tests/test_gh_budget_wait.py (27) pass locally; git diff --check is clean. Fleet fast-check is being rerun at the rebased head.

Agent session: agent session
🤖 Generated with Claude Code

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_ci_nightly_off_blacksmith.py">

<violation number="1" location="tests/test_ci_nightly_off_blacksmith.py:186">
P2: This assertion does not verify that `prepare-nightly-variant` is a distinct placement token: the primary job name satisfies the substring check by itself. Use token membership so the debt test catches the retry job being omitted from `JOBS`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread tests/test_ci_nightly_off_blacksmith.py Outdated
place = next(step for step in jobs["build-nightly-app"]["steps"] if step.get("id") == "place-light")
place_jobs = place["env"]["JOBS"]
self.assertIn("prepare-nightly-variant-primary", place_jobs)
self.assertIn("prepare-nightly-variant", place_jobs)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This assertion does not verify that prepare-nightly-variant is a distinct placement token: the primary job name satisfies the substring check by itself. Use token membership so the debt test catches the retry job being omitted from JOBS.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_nightly_off_blacksmith.py, line 186:

<comment>This assertion does not verify that `prepare-nightly-variant` is a distinct placement token: the primary job name satisfies the substring check by itself. Use token membership so the debt test catches the retry job being omitted from `JOBS`.</comment>

<file context>
@@ -164,16 +163,28 @@ def test_prepare_runs_on_linux_and_executes_no_mach_o(self) -> None:
+        place = next(step for step in jobs["build-nightly-app"]["steps"] if step.get("id") == "place-light")
+        place_jobs = place["env"]["JOBS"]
+        self.assertIn("prepare-nightly-variant-primary", place_jobs)
+        self.assertIn("prepare-nightly-variant", place_jobs)
+        self.assertIn("attest-nightly-remote-daemon", place_jobs)
         moved_out = ("install-cmux-tui-client.sh", "strip-release-bundle.sh", "write-cmux-tui-version.sh",
</file context>
Suggested change
self.assertIn("prepare-nightly-variant", place_jobs)
self.assertIn("prepare-nightly-variant", place_jobs.split())

Comment thread .github/workflows/nightly.yml Resolved
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review e148c73: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30113 -f ref=lane/nightly-prepare-attest-routing -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base bcaf263; does not block

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Found 9 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpw0p_6ptk/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpx_vdim8f/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_no_suite_runs_page_javascript_on_a_built_bundle (main.BundleInputs.test_no_sui
te_runs_page_javascript_on_a_built_bundle)/
test_no_suite_runs_page_javascript_on_a_built_bundle (main.BundleInputs.test_no_sui
te_runs_page_javascript_on_a_built_bundle)
View Logs
the committed export failed: check-daemon-capabilities-fresh: plans/cmux-next/
daemon-capabilities.json is stale:
View Logs
the committed export failed: check-daemon-capabilities-fresh: plans/cmux-next/
daemon-capabilities.json is stale:
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 7e5f23f

BLOCK at 7e5f23f

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] Adding the three new placement readers makes tests/test_nightly_trusted_routing.py:501 fail because its LIGHT set still contains only the original two jobs. This test is run by ci-guards.yml:1588.

  2. [P1] [blocking] Adding build-nightly-app to the attestation job’s needs breaks tests/test_gh_budget_wait.py:408, which is an explicit CI guard at ci-guards.yml:1449. The targeted PR test passes, but this required guard fails.

  3. [P1] [blocking] .github/workflows/nightly.yml:3023 places attest-nightly-remote-daemon on the owned mini route even though it has id-token: write and attestations: write at lines 3095–3096. Owned minis are persistent runners that also execute portable/untrusted jobs; exposing the attestation OIDC credential there crosses the runner trust boundary. Keep this attestation on the existing isolated overflow runner.

  4. [P2] [follow-up] The new placement list at .github/workflows/nightly.yml:3019-3024 does not set NIGHTLY_WAIT_FOR_FLEET=1, and the step still has a two-minute timeout. portable_job_placement.py only enables the three-minute wait when that variable is set (and needs roughly five minutes including its read deadline), so these jobs immediately fall back instead of using the claimed bounded fleet window.

Validation: python3 tests/test_ci_nightly_off_blacksmith.py passed (16 tests); the trusted-routing and gh-budget guard tests fail as described.

Follow-ups:

  • [P2] [follow-up] Enable and correctly budget the bounded fleet wait for the light placement step.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 7e5f23f

BLOCK at 7e5f23f

Visual verdict: no images; Leo tries the build

  1. [P1][blocking] Adding three more light_mini consumers at .github/workflows/nightly.yml:3023 leaves tests/test_nightly_trusted_routing.py:501 asserting that only the original two jobs read placement. python3 tests/test_nightly_trusted_routing.py now fails, so the CI nightly validation is red.

  2. [P1][blocking] The new dependency at .github/workflows/nightly.yml:3090 leaves tests/test_gh_budget_wait.py:408 expecting the old needs list. python3 tests/test_gh_budget_wait.py fails on that assertion.

  3. [P2][follow-up] .github/workflows/nightly.yml:3021-3024 does not set NIGHTLY_WAIT_FOR_FLEET=1. portable_job_placement.py only enables its 180-second wait with that variable, so this placement immediately falls back to Blacksmith instead of using the promised three-minute fleet window.

Follow-ups:

  • [P2][follow-up] Enable NIGHTLY_WAIT_FOR_FLEET=1 for the light placement and give the step enough timeout for that wait.

@teamleaderleo
teamleaderleo force-pushed the lane/nightly-prepare-attest-routing branch from 7e5f23f to 80e16f8 Compare November 1, 2026 04:33
@teamleaderleo
teamleaderleo force-pushed the lane/nightly-prepare-attest-routing branch from 80e16f8 to ad1a985 Compare November 1, 2026 04:35
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at ad1a985

BLOCK at ad1a985

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] .github/workflows/nightly.yml:3310 — Both preparation jobs now support macOS but lack entries in .github/runners.json. This breaks test_runner_routing_table.py, which CI runs. Restoring the previous runner expressions makes the check pass. Register both jobs as portable-checks.

  2. [P1] [blocking] .github/workflows/nightly.yml:3025 — NIGHTLY_WAIT_FOR_FLEET=1 also enables unconditional fleet placement when capacity reads fail or the snapshot is stale. Reproduced with an offline runner and missing snapshot: all four jobs select the owned pool immediately, with zero waiting and no Blacksmith overflow. This regresses the existing daemon/tree-marker routes and can stall nightly publication; the rescue watcher excludes these side-pool jobs. Preserve bounded overflow when capacity is unknown.

The three focused suites passed: 73 tests.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/nightly.yml">

<violation number="1" location=".github/workflows/nightly.yml:3023">
P2: `attest-nightly-remote-daemon` is no longer routed through placement: it is omitted from `JOBS` and its runner expression has no mini-placement fallback, so it always uses Blacksmith on the owned repository. Keep the attestation job in placement and retain a dependency/output path that lets it read the placement result.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/nightly.yml Outdated
timeout-minutes: 5
env:
JOBS: build-nightly-remote-daemon await-nightly-tree-marker
JOBS: build-nightly-remote-daemon prepare-nightly-variant-primary prepare-nightly-variant await-nightly-tree-marker

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: attest-nightly-remote-daemon is no longer routed through placement: it is omitted from JOBS and its runner expression has no mini-placement fallback, so it always uses Blacksmith on the owned repository. Keep the attestation job in placement and retain a dependency/output path that lets it read the placement result.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 3023:

<comment>`attest-nightly-remote-daemon` is no longer routed through placement: it is omitted from `JOBS` and its runner expression has no mini-placement fallback, so it always uses Blacksmith on the owned repository. Keep the attestation job in placement and retain a dependency/output path that lets it read the placement result.</comment>

<file context>
@@ -3018,10 +3018,11 @@ jobs:
+        timeout-minutes: 5
         env:
-          JOBS: build-nightly-remote-daemon prepare-nightly-variant-primary prepare-nightly-variant attest-nightly-remote-daemon await-nightly-tree-marker
+          JOBS: build-nightly-remote-daemon prepare-nightly-variant-primary prepare-nightly-variant await-nightly-tree-marker
           NIGHTLY: "1"
+          NIGHTLY_WAIT_FOR_FLEET: "1"
</file context>

@teamleaderleo
teamleaderleo force-pushed the lane/nightly-prepare-attest-routing branch from ad1a985 to 45793e9 Compare November 1, 2026 04:41

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/ci/portable_job_placement.py Resolved
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 45793e9

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 45793e9

Visual verdict: no images; Leo tries the build

  1. [P2][follow-up] .github/workflows/nightly.yml:3023,3088-3093 — attest-nightly-remote-daemon is not included in place-light and still has a fixed Blacksmith runner. The PR description and commit title claim attestation is routed through fleet placement, so this remains an unmet part of the stated change.

Targeted placement, routing, and nightly tests pass.

Follow-ups:

  • [P2][follow-up] Route attest-nightly-remote-daemon as described, or correct the PR scope/title.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 4b98238

BLOCK at 4b98238

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] portable_job_placement.py:165 — Removing the unknown-capacity fallback also changes nightly_linux_admission.routes(): when admission is unavailable, early controls now select cmux-fleet-linux-x64 instead of the configured owned minis. This breaks test_missing_admission_keeps_controls_on_existing_owned_routes, which runs in ci-guards.yml. Reproduced with python3 -m unittest discover -s tests -p 'test_nightly_linux_admission.py'; restoring only the removed branch makes all 24 tests pass. Scope the new overflow behavior to preparation placement while preserving the admission caller’s fallback contract.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo
teamleaderleo force-pushed the lane/nightly-prepare-attest-routing branch from 0561328 to 856c1ed Compare November 1, 2026 05:31
@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 856c1ed

BLOCK at 856c1ed

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] .github/workflows/nightly.yml:3393: The new runner expressions break test_prepare_runs_on_linux_and_executes_no_mach_o, whose assertions still require Blacksmith-only routes. test_ci_nightly_off_blacksmith.py fails locally and runs in the release-notary CI guard. Update both preparation assertions to cover the new routes.

  2. [P2] [follow-up] .github/workflows/nightly.yml:3841: When both preparation jobs select minis, the fallback reuses the primary’s pool even after primary failure. If that pool went offline, the retry queues there instead of taking the previous Blacksmith failover. Refresh placement for the retry or restrict mini reuse to primary success.

  3. [P2] [follow-up] scripts/ci/portable_job_placement.py:258: Freshness is checked only before polling. A fresh initial read followed by unread inventory and snapshot still produces seven reads and 180 seconds of sleeps in a mocked reproduction. Apply the freshness check after every read so newly unknown capacity triggers immediate overflow.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at f70b3dd

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at f70b3dd

Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] scripts/ci/portable_job_placement.py:258 — Freshness is checked only before the polling loop. If a subsequent read fails or returns a stale snapshot, polling continues until the deadline instead of immediately falling through to Blacksmith. A mocked failed first refresh reproduced another 150 seconds of waiting. Recheck freshness after every read and exit the loop when capacity becomes unknown.

All eight targeted test suites passed (172 tests).

Follow-ups:

  • Recheck capacity freshness during polling and add coverage for failed or stale refreshes.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves f70b3dd: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · took over from an earlier session · session

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (gpt-6-astra high): SHIP at e148c73

This branch was previously deployed

1 inactive deployment
ci-routing-agent — e148c732 Deployed Nov 1, 2026 by teamleaderleo via wait for same-key tree publisher #94850
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant