Skip to content

ci(nightly): the all-nightly reservation's idle runners take nightly-next's light jobs - #31410

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-light-placement
Nov 1, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-light-placement

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

"Place the light jobs on a free mini", at the end of build-nightly-app, only offers glaeda-side-* labels. The all-nightly reservation's light runners (glaeda-nightly-only-m4pro, four per dedicated Mac) exist for these jobs, but they never qualified. So on every nightly-next run the step waited its full three minutes on busy side minis, re-reading runners every 30 s, and then sent prepare-nightly-variant-primary, prepare-nightly-variant and await-nightly-tree-marker to Blacksmith anyway.

That wait sits on the path from landing to feed, and the rule allows 30 minutes for that path (the feed was about 45 min behind tonight).

Change (scripts/ci/portable_job_placement.py and the step's env in nightly.yml):

  • New NIGHTLY_ONLY_LABEL (vars.CI_NIGHTLY_ONLY_RUNNER). On a nightly placement whose runners were read, attempt 1 only (the same attempt rule, factored into attempt_refusal), the jobs take the reservation's idle runners at once.
  • Idle runners are counted less the jobs the snapshot shows queued on that label. The warm builder carries only glaeda-nightly-only-m4pro-builder, so it is never counted.
  • When the reservation is unset, busy or unread, or on a later attempt, the existing side, wait and Blacksmith logic runs unchanged.
  • The three jobs use no secrets and no environment, and the reservation already admits nightly.yml jobs on nightly-next (decide, resolve and "build nightly inputs" run there today).

Performance

metric before after delta harness or run link
"Place the light jobs on a free mini" duration 186 s (runs 41354494707, 41355147140, 41356701375) under 15 s (run 41360732074: the three jobs took glaeda-nightly-only-m4pro) about -3 min per build build-nightly-app step timings
runner reads in that step about 42 REST requests (github_spend) 4 requests (run 41360732074) about -38 requests per run github_spend lines

Tests:

  • tests/test_ci_portable_job_placement.py: 36 pass, including new NightlyOnly cases for placement, the queue count, never the builder, the busy, unread, unset and later-attempt fallbacks, and the workflow wiring.
  • tests/test_ci_nightly_off_blacksmith.py: 21 pass.
  • test_ci_linux_single_size fails the same way on base (cmux-next-base-tier-kick, cmux-next-dev-kick, ios-internal-nightly-trigger).

Agent session: agent session

🤖 Generated with Claude Code

…next's light jobs

Place the light jobs on a free mini only offered glaeda-side-* labels, so on
the all-nightly route every nightly-next run waited its full three minutes on
busy side minis and then sent prepare-nightly-variant(-primary) and
await-nightly-tree-marker to Blacksmith: 186 s at the end of build-nightly-app
in runs 41354494707, 41355147140 and 41356701375, on the feed's critical path,
with a runner read every 30 s. While the reservation (vars.CI_NIGHTLY_ONLY_RUNNER)
has idle runners, those jobs take it at once; otherwise placement is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 226ac57: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr31410 -f ref=lane/nightly-light-placement -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on c4a9ee3 to compare; do not merge until green; see lane-rules

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 226ac57

  • When the reservation has room, all three light jobs are placed on the first pass and the loop exits.
  • When runners can't be read, the new branch is skipped and the old path runs.
  • The runs-on wiring (light_mini_runner, light_mini_jobs, light_attempt) works with the reservation label, and the warm builder label is never counted.
  • The three jobs use no secrets and no environment, with contents: read.
  • Passing: placement, nightly_off_blacksmith, nightly_trusted_routing, runner_label_policy, runner_routing_table, fork_runner_routing, feat_ci_guard_group, route_refusal_detail, nightly_push_throttle.
  • test_ci_self_hosted_guard.sh and test_ci_linux_single_size fail the same way on base.

Follow-ups:

  • [follow-up] With only partial room, the wait loop can switch between the reservation and the side route on later passes. Keep reservation placements once made, and let the side route take only what's left.
  • [follow-up] read() only reads runners when eligible() passes, which needs SIDE_LABEL to be an owned side label. Let it also read when NIGHTLY=1 and NIGHTLY_ONLY_LABEL is set.
  • [follow-up] Pin NIGHTLY_ONLY_LABEL to nightly_build_runner.NIGHTLY_ONLY_LABEL, and check that the janitor snapshot has a pools entry for that label.

@teamleaderleo
teamleaderleo merged commit cb4f169 into feat-cmux-next Nov 1, 2026
168 of 179 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-light-placement branch November 1, 2026 23:06
teamleaderleo added a commit that referenced this pull request Nov 1, 2026
…#31463)

prepare-nightly-variant-primary moves the whole app twice and scans every file
of it. Placed on a dedicated nightly Mac by #31410 it took 4.8 min (run
41360732074: download 59 s, browser-data proof 99 s, upload 62 s) against 57 s
on its Blacksmith route (run 41358657711), which undid the 3 min #31410 saved.
The two tiny light jobs stay on the reservation; the primary is no longer
offered, so it neither waits nor lands on a Mac.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 226ac57c Deployed Nov 1, 2026 by teamleaderleo via wait for same-key tree publisher #97782
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant