Skip to content

ci(nightly): prefer dedicated mini for variant preparation - #30287

Closed
teamleaderleo wants to merge 6 commits into
feat-cmux-nextfrom
lane/nightly-dedicated-variant
Closed

teamleaderleo wants to merge 6 commits into
feat-cmux-nextfrom
lane/nightly-dedicated-variant

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Leo said: “BUT WE HAVE A DEDICATED F***ING MAC FOR THAT.”

This stacked follow-up gives only prepare-nightly-variant-primary and prepare-nightly-variant an opt-in preference for the dedicated M4 Pro mini. The selector is deliberately exact: repository manaflow-ai/cmux, NEXT track, attempt 1, and CI_NIGHTLY_ONLY_RUNNER == glaeda-nightly-only-m4pro. When the variable is unset (the current state until rollout and hook registration), existing light-placement and Linux overflow routes are unchanged. Forks, retries, other tracks, and non-matching values preserve their existing fallbacks.

This PR is stacked on #30113 at f70b3ddbddcefb359e6484276b977c3c47e51517; it does not modify #30113. After #30113 lands, rebase this single dedicated-routing commit onto the new feat-cmux-next tip before the pinned merge.

Validation:

  • python3 tests/test_nightly_dedicated_variant.py (4 passed)
  • python3 tests/test_nightly_trusted_routing.py (34 passed)
  • python3 tests/test_ci_portable_job_placement.py (30 passed)
  • python3 tests/test_ci_nightly_off_blacksmith.py passed
  • YAML parse and git diff --check passed

Agent session: https://cmux.com/deeplink/agent?id=aid-cd7a5a0ee7f7c79a3232
🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Makes prepare-nightly-variant-primary and prepare-nightly-variant prefer a dedicated M4 Pro mini runner during NEXT-track nightly preparation. The routing is opt-in: it only applies on attempt 1 for manaflow-ai/cmux when CI_NIGHTLY_ONLY_RUNNER is set to glaeda-nightly-only-m4pro; all other forks, retries, tracks, and unset values keep the existing light-placement and Linux overflow fallbacks.

Adds tests covering the reservation selector, including assertion that only the two preparation jobs read it and that unset or non-matching contexts keep existing fallbacks.

  • Allowlists CI_NIGHTLY_ONLY_RUNNER in the runner label policy: only the exact glaeda-nightly-only-m4pro label is accepted, and the variable counts as a no-hosted-Mac variable.
  • Propagates the variable through the health-report and repo-variables workflows so drift detection can check it.

Written for commit 28d33c8. Summary will update on new commits.

View guided diff Turn on auto-fix

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_nightly_dedicated_variant.py">

<violation number="1" location="tests/test_nightly_dedicated_variant.py:1">
P2: This new test file is not registered in `tests/test-execution.toml`, so the registry validation fails for this PR. No workflow runs it either, so these assertions would never execute in CI. Add a `[[test]]` entry for `tests/test_nightly_dedicated_variant.py` and wire it into a job that runs, like `test_nightly_trusted_routing.py` in `ci-guards.yml`.</violation>
</file>

<file name=".github/workflows/nightly.yml">

<violation number="1" location=".github/workflows/nightly.yml:3847">
P2: With the reservation enabled, the fallback `prepare-nightly-variant` also routes to the single mini after a successful primary. There it is a no-op that still queues behind other variants' primary legs, which reintroduces the queued hop the job comment warns about. Gate the reservation on the primary having failed, so a successful primary keeps the existing fallbacks.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/nightly.yml Resolved
@@ -0,0 +1,76 @@
#!/usr/bin/env python3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This new test file is not registered in tests/test-execution.toml, so the registry validation fails for this PR. No workflow runs it either, so these assertions would never execute in CI. Add a [[test]] entry for tests/test_nightly_dedicated_variant.py and wire it into a job that runs, like test_nightly_trusted_routing.py in ci-guards.yml.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_dedicated_variant.py, line 1:

<comment>This new test file is not registered in `tests/test-execution.toml`, so the registry validation fails for this PR. No workflow runs it either, so these assertions would never execute in CI. Add a `[[test]]` entry for `tests/test_nightly_dedicated_variant.py` and wire it into a job that runs, like `test_nightly_trusted_routing.py` in `ci-guards.yml`.</comment>

<file context>
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""The dedicated nightly-only mini is opt-in for the two preparation legs."""
+
</file context>

runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-32vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
# Keep the exact reservation contract from the primary leg. It is unset
# until the dedicated mini's rollout and hook registration complete.
runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.repository == 'manaflow-ai/cmux' && needs.decide.outputs.track == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-32vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With the reservation enabled, the fallback prepare-nightly-variant also routes to the single mini after a successful primary. There it is a no-op that still queues behind other variants' primary legs, which reintroduces the queued hop the job comment warns about. Gate the reservation on the primary having failed, so a successful primary keeps the existing fallbacks.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 3847:

<comment>With the reservation enabled, the fallback `prepare-nightly-variant` also routes to the single mini after a successful primary. There it is a no-op that still queues behind other variants' primary legs, which reintroduces the queued hop the job comment warns about. Gate the reservation on the primary having failed, so a successful primary keeps the existing fallbacks.</comment>

<file context>
@@ -3838,7 +3842,9 @@ jobs:
-    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-32vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
+    # Keep the exact reservation contract from the primary leg. It is unset
+    # until the dedicated mini's rollout and hook registration complete.
+    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.repository == 'manaflow-ai/cmux' && needs.decide.outputs.track == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-32vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
     timeout-minutes: 100
     strategy:
</file context>
Suggested change
runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.repository == 'manaflow-ai/cmux' && needs.decide.outputs.track == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-32vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.prepare-nightly-variant-primary.result != 'success' && github.repository == 'manaflow-ai/cmux' && needs.decide.outputs.track == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || needs.build-nightly-app.outputs.light_attempt == format('{0}', github.run_attempt) && contains(needs.build-nightly-app.outputs.light_mini_jobs, ' prepare-nightly-variant ') && needs.build-nightly-app.outputs.light_mini_runner || needs.prepare-nightly-variant-primary.result == 'success' && (vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404') || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

View guided diff | Turn on auto-fix | Re-trigger cubic

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, review-box): SHIP at 28d33c8

No findings. CI-only diff; 161 targeted tests passed. Broader guard failures match the merge base or stem from omitted certificate fixtures.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/ci/runner_label_policy.py">

<violation number="1" location="scripts/ci/runner_label_policy.py:419">
P3: This trims the value before enforcing the exact selector, so a whitespace-padded label passes the variable policy while both jobs’ raw equality checks fall through to existing routes. Pass the untrimmed value here so non-exact configuration is reported.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

elif name == NIGHTLY_NEXT_BUILD_VARIABLE:
reason = nightly_next_build_reason(value.strip())
elif name == NIGHTLY_ONLY_VARIABLE:
reason = nightly_only_reason(value.strip())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This trims the value before enforcing the exact selector, so a whitespace-padded label passes the variable policy while both jobs’ raw equality checks fall through to existing routes. Pass the untrimmed value here so non-exact configuration is reported.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/runner_label_policy.py, line 419:

<comment>This trims the value before enforcing the exact selector, so a whitespace-padded label passes the variable policy while both jobs’ raw equality checks fall through to existing routes. Pass the untrimmed value here so non-exact configuration is reported.</comment>

<file context>
@@ -399,6 +415,8 @@ def drifted_runner_variables(
         elif name == NIGHTLY_NEXT_BUILD_VARIABLE:
             reason = nightly_next_build_reason(value.strip())
+        elif name == NIGHTLY_ONLY_VARIABLE:
+            reason = nightly_only_reason(value.strip())
         elif name == FAST_LANE_VARIABLE:
             reason = fast_lane_reason(value.strip())
</file context>
Suggested change
reason = nightly_only_reason(value.strip())
reason = nightly_only_reason(value)

Base automatically changed from lane/nightly-prepare-attest-routing to feat-cmux-next November 1, 2026 09:17
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Merge conflicts with feat-cmux-next, not resolved: base already landed most of this plumbing (#31266, #31291 and earlier). CI_NIGHTLY_ONLY_RUNNER is already in ci-health-report.yml and ci-repo-variables.yml, runner_label_policy.py already validates it (so the new nightly_only_reason branch would be dead code), and test_ci_self_hosted_guard.sh already defines nightly_only_selector (redefining it would change the guard). Only the prepare-nightly-variant(-primary) routing to the dedicated mini is new, so this needs a small rework on current base rather than a merge.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 13 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in .github/runners.json, .github/workflows/nightly.yml, scripts/ci/runner_label_policy.py, tests/test_ci_nightly_off_blacksmith.py, tests/test_ci_portable_job_placement.py, tests/test_ci_self_hosted_guard.sh, tests/test_nightly_trusted_routing.py. Listed for a lane to resolve.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Closing as superseded; nothing left to rework onto feat-cmux-next.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 28d33c88 Deployed Nov 1, 2026 by teamleaderleo via cmux-tui tree preflight #94388
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant