Repository navigation
ci: keep queued nightly controls on the fleet - #30108
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_ci_nightly_off_blacksmith.py">
<violation number="1" location="tests/test_ci_nightly_off_blacksmith.py:96">
P2: This new assertion pins the classic `nightly` track to `blacksmith-16vcpu-ubuntu-2404` through `LINUX_RUNNER`. That conflicts with the repo rule that nightly jobs never take Blacksmith labels, and the new test now enforces that route. If the classic track is intentionally out of scope, drop this assertion so the test does not lock in Blacksmith routing.</violation>
</file>
<file name=".github/workflows/nightly.yml">
<violation number="1" location=".github/workflows/nightly.yml:3153">
P2: With the new fleet branch ahead of the Blacksmith tail, these two jobs no longer fall back to Blacksmith when no fleet runner starts. Confirm that the 180-second overflow in `portable_job_placement.py` still reaches them, or add a fallback that the overflow switch can change.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| with self.subTest(job=name, attempt=attempt, placement=placement_attempt): | ||
| self.assertEqual(evaluate(expression, context), "cmux-fleet-linux-x64") | ||
| context["needs.decide.outputs.track"] = "nightly" | ||
| self.assertEqual(evaluate(expression, context), "blacksmith-16vcpu-ubuntu-2404") |
There was a problem hiding this comment.
P2: This new assertion pins the classic nightly track to blacksmith-16vcpu-ubuntu-2404 through LINUX_RUNNER. That conflicts with the repo rule that nightly jobs never take Blacksmith labels, and the new test now enforces that route. If the classic track is intentionally out of scope, drop this assertion so the test does not lock in Blacksmith routing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_nightly_off_blacksmith.py, line 96:
<comment>This new assertion pins the classic `nightly` track to `blacksmith-16vcpu-ubuntu-2404` through `LINUX_RUNNER`. That conflicts with the repo rule that nightly jobs never take Blacksmith labels, and the new test now enforces that route. If the classic track is intentionally out of scope, drop this assertion so the test does not lock in Blacksmith routing.</comment>
<file context>
@@ -78,6 +78,33 @@ def code_lines(path: Path):
+ with self.subTest(job=name, attempt=attempt, placement=placement_attempt):
+ self.assertEqual(evaluate(expression, context), "cmux-fleet-linux-x64")
+ context["needs.decide.outputs.track"] = "nightly"
+ self.assertEqual(evaluate(expression, context), "blacksmith-16vcpu-ubuntu-2404")
+ context["github.repository_owner"] = "fork-owner"
+ self.assertEqual(evaluate(expression, context), "ubuntu-24.04")
</file context>
| needs: decide | ||
| if: needs.decide.outputs.should_build == 'true' && (github.event_name != 'schedule' || github.event.schedule == '47 8 * * *') | ||
| runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_linux_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_linux_runner || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_mini_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_mini_runner || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404' }} | ||
| runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_linux_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_linux_runner || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_mini_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_mini_runner || needs.decide.outputs.track == 'nightly-next' && 'cmux-fleet-linux-x64' || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404' }} |
There was a problem hiding this comment.
P2: With the new fleet branch ahead of the Blacksmith tail, these two jobs no longer fall back to Blacksmith when no fleet runner starts. Confirm that the 180-second overflow in portable_job_placement.py still reaches them, or add a fallback that the overflow switch can change.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 3153:
<comment>With the new fleet branch ahead of the Blacksmith tail, these two jobs no longer fall back to Blacksmith when no fleet runner starts. Confirm that the 180-second overflow in `portable_job_placement.py` still reaches them, or add a fallback that the overflow switch can change.</comment>
<file context>
@@ -3150,7 +3150,7 @@ jobs:
needs: decide
if: needs.decide.outputs.should_build == 'true' && (github.event_name != 'schedule' || github.event.schedule == '47 8 * * *')
- runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_linux_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_linux_runner || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_mini_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_mini_runner || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404' }}
+ runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_linux_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_linux_runner || needs.decide.outputs.early_attempt == format('{0}', github.run_attempt) && contains(needs.decide.outputs.early_mini_jobs, ' resolve-nightly-cmux-tui-client ') && needs.decide.outputs.early_mini_runner || needs.decide.outputs.track == 'nightly-next' && 'cmux-fleet-linux-x64' || vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404' }}
permissions:
contents: read
</file context>
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base c06e62a; does not block Shipped in:
|
GitHub Actions
Inactive
|
Found 1 test failure on Blacksmith runners: Failure
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
|
Review (codex gpt-6-astra high): SHIP at 60f6989 Diff is only the two NEXT resolver/recovery fallback selectors plus tests. Precedence, fork/track guard and no-secrets are all correct; 65 focused tests and diff-check pass. The existing self-hosted guard failure is an unrelated base red. Its HOLD prerequisite, HQ#5804 (gh on GCP JIT startup), merged as 80ada0f8. |
|
Review (codex gpt-6-astra high): SHIP at 60f6989. Focused routing, nightly, trusted and fork tests pass; no blocking finding. |
|
Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 60f6989 BLOCK at 60f6989 Visual verdict: no images; Leo tries the build
Validation: all 97 relevant routing, placement, nightly policy, and fork tests passed. These do not exercise the runner image. |
|
Owned by an agent session · session |
…inder call it The fleet-linux fallback can land these jobs on a GCP JIT VM that predates HQ#5804's startup script and has no gh; find-nightly-next-recovery.py then crashes with FileNotFoundError. Install gh (distro, then cli.github.com apt repo) before the budget wrapper, and fail with a clear error if it can't. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189
GitHub Actions
Inactive
|
Review (codex, router-review): SHIP at d5c773b Reviewer: codex gpt-6.1-sol xhigh, fast tier. SHIP at d5c773b Visual verdict: no images; Leo tries the build No correctness, regression, security, or test failures found in the changed paths. Focused routing, nightly policy, YAML, and shell checks pass. Follow-ups: none |
NEXT nightly run 41269080663 queued its TUI resolver and recovery finder on Blacksmith 16 vCPU while no runner started. When their early placement is empty or stale, these two secret-free jobs now request the existing GCP JIT fleet pool. Valid current-attempt fleet Linux and owned-mini placements still take precedence. Other tracks and fork routing keep their existing behavior.
Leo: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT”.
This is the immediate two-job repair requested ahead of the larger admission change. It never treats missing placement as permission for paid work. The recorded 180-second no-start overflow mechanism remains in #30085. This change does not cancel or rerun an existing nightly; a newer source can build alongside it. No signing job, environment, credential, or permission changes.
Prior art: #29864 supplies early placement and #30042 preserves fleet placement when the routing API quota fails. Their empty-output fallback still selected Blacksmith; this fixes that last branch. #30085 is this lane's held admission implementation. #30081 and #30098 own signing and publication routing and are unchanged.
Validation: 65 routing, nightly policy and fork tests pass. Both new default-route assertions fail against base efbe429 (Blacksmith 16 instead of fleet Linux). The shell guard's eight failing selectors are byte-for-byte identical to base; neither changed selector is flagged. These jobs do no compilation, so compilation caches are not involved.
Performance
Changelog
none
Agent session: agent session
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Keeps the two nightly-next control jobs (
resolve-nightly-cmux-tui-clientandfind-nightly-next-recovery) on the fleet when their early placement is empty or stale, instead of letting them fall through to Blacksmith.runs-onexpression so an empty or stale placement plus thenightly-nexttrack resolves tocmux-fleet-linux-x64.HQ#5804 (GCP JIT startup installs
gh) has merged. As a fail-safe for VMs made before it, both jobs now runscripts/ci/nightly-steps/ensure-gh.shon Linux before their firstghcall: it installsghfrom apt, falls back to the cli.github.com apt repo, and fails the step with a clear error rather than letting the finder crash with FileNotFoundError.Written for commit 60f6989. Summary will update on new commits.