[teamleaderleo]

An older nightly-next build hands its sign slot to a newer build whose sign job is already queued, running or signed. Nothing is cancelled, and the newest finished build always signs.

What was wrong

GitHub gives a free runner the oldest queued job on its label, and a runner can't choose another. Publication is latest-wins. So on 2026-11-04 around 17:25Z, the three sign slots were held by the oldest nightly-next builds (runs 41866781601, 41867447845, 41868557041), and the newer builds' build-sign-notarize-nightly jobs waited behind them: 15, 19 and 24 minutes for the three that had started, with three more still queued (41869470224, 41870869803, 41873130397). Each older build then published a nightly that the next one replaced within minutes, and the newest tip reached the feed about an hour after its landing.

The existing "Leave notarization to a newer published build" check (sign-pick-overtaken.sh, #33109) runs in pick-nightly-sign-runner, before the sign job joins the queue. With oldest-first order it almost never fires.

What this changes

• One step, Yield the sign slot to a newer build, runs in build-sign-notarize-nightly right after the claim, before any secret or signing work. It is scripts/ci/nightly_sign_yield.py, run from the trusted signing controls.
  • It reads the in-progress nightly-next push runs newer than this one.
  • If one has its own sign job queued, running or signed, this job exits 1 with yielded=true, and the runner takes the next queued job (about a minute instead of a full sign and notarize).
  • Otherwise, and on any doubt (an unreadable answer, a malformed field, a listing that doesn't add up), it signs as before.
  • Scope: attempt 1 of a nightly-next push only. Re-runs, dispatches and main's nightly never yield. Setting NIGHTLY_NEXT_SIGN_YIELD=0 turns it off.
• report-nightly-failure does not report a yielded sign whose run had nothing else fail. This is the same pattern as its existing "replaced" build.
• Nothing is cancelled, and no concurrency group changes. The rulebook's "never cancel a started NEXT nightly" holds. Verify, publish and deltas are skipped for the yielded run because the sign job did not succeed, as for any failed sign.

The rule this narrows (#33109)

sign-pick-overtaken.sh says: "Only a published newer build counts, never a newer push alone: a newer run that then fails must not leave the feed with nothing (#28961)."

• What still holds: a newer push alone still never counts. A newer run whose build is still running, or whose sign job failed, was skipped or was cancelled, is no reason to yield.
• What now counts: a newer run whose sign job exists, which GitHub creates only once that run's build jobs pass.
• Why it can't starve the feed: the newest such sign job never yields (no newer run has one), so the newest finished build always signs. ci(nightly): nightly-next runs stop skipping their own builds; no 32 vCPU pins; display visual gate opt-in #28961's starvation came from a run skipping its build as soon as nightly-next moved, before any newer build existed.
• What is given up: if that newest sign then fails, the older builds that yielded to it go unsigned, and the next tip builds as usual.

Decided on cmuxterm-hq#7079. Ordering the slots newest-first was tried first. teamleaderleo/glaeda#1815 found that a runner can't choose which queued job it takes; glaeda can only refuse a job, which fails it.

Tests

tests/test_nightly_sign_yield.py (new, linux-guard):

• the 2026-11-04 shape: the newest queued build never yields, nor does a newer run whose sign job doesn't exist yet;
• an older build yields to a newer sign job that is queued, running or signed;
• a newer push alone never counts (no sign job, a failed, skipped or cancelled one, or the legacy-identity signer);
• an older run is never a reason;
• out of scope (attempt 2, dispatch, main, another track, NIGHTLY_NEXT_SIGN_YIELD=0, a bad id): it signs without a read;
• any doubt signs;
• the CLI writes yielded=false and exits 0 when the API is unreachable;
• wiring: the step sits right after the claim, with no secret before it or in it, and report-nightly-failure skips only a yielded sign.

Red on base: the wiring tests fail on 4d20a29 (no such step or output); the decision tests have no script to load there. Green on this head.

python3 -m unittest tests/test_nightly_sign_yield.py tests/test_nightly_step_coverage.py: 14 OK on this head. Every other test file that reads nightly.yml (106) was run on the head and on 4d20a29: 69 pass on both, and the remaining 37 fail exactly the same tests on both (they need tools this checkout lacks, such as actionlint).

Observed / Not run

• Observed: the run and job timings above, read from the Actions API at about 17:25Z.
• Not run: a live nightly-next run with the step. The first nightly-next runs after this lands show it in their sign job's log (signing: ... or the Sign yield notice).

Follow-up, not here: a superseded run's sign-nightly-identity-legacy also takes the exclusive class on its mini (35 min on run 41799796683); the same yield could apply to it.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

An older nightly-next build now hands its sign slot to a newer build whose sign job is queued, running, or signed, cutting the newest tip's feed delay from roughly 20 minutes per slot to about a minute.

GitHub gives a free runner the oldest queued job on its label, so the newest build's sign job waited behind older builds whose nightlies were replaced within minutes. A new step right after the signing claim, before any secret, checks the in-progress nightly-next runs; if a newer run's sign job already exists (GitHub creates it only once that run's build passes), this job exits 1 and the runner picks the next queued job. Nothing is cancelled, and report-nightly-failure skips a yielded sign whose run had nothing else fail.

• Narrowing nightly: spend no notarization on a build a newer published build is above #33109's rule: a newer push alone still never counts, but a newer run whose sign job exists now does. The newest sign job never yields, so the newest finished build always signs; the trade-off is that if that sign then fails, the older builds that yielded to it go unsigned.
• The yield script pages the run and job listings until an empty page, since a status-filtered list's total_count can run ahead; more than MAX_PAGES pages signs as before. The test is registered as manual, so the registry validator passes.

^{Written for commit a35ac85. Summary will update on new commits.}

[teamleaderleo]

Owned by an agent session · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review a35ac85: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr35752 -f ref=nightly-sign-yield-newer-queued -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4190094575701 (036f8aa538)
• cmux NEXT nightly 1.0.0-nightly.4191131679101 (e01bbae5c9)

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
main.R2UploadRequestsTests.test_a_slow_large_put_is_bounded_by_idle_gaps_not_total_ time/test_a_slow_large_put_is_bounded_by_idle_gaps_not_total_time	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (goal-nightly-30min, independent of the author): BLOCK at bf25b77

1. Blocking, a red of this PR's own: tooling-tests. tests/test-execution.toml registers tests/test_nightly_sign_yield.py as lane = "linux-guard", but no workflow step runs it. validate_test_execution_registry.py fails with "linux-guard lane is not run by any workflow". base_red_excuse.py doesn't excuse it, because base run 41883647299 is green. ci: live-admit and rescue nightly-next controls #30657 went red the same way. Lawrence's 2026-10-23 rule is no new CI tests, so the fix is lane = "manual" with a reason (see the neighbouring entries).

The logic checks out:

• The yield step runs after the claim and before any secret. The pre-yield steps succeed, so owned_pool_rescue.refused() won't rerun a yielded job.
• feed-watch's legacy_rerun reruns only published builds, and a yielded build is never published. Verdicts come only from feed builds, so a yield doesn't mark a commit bad.
• Sign jobs request the shared glaeda-nightly-std-xcode-26.6 label with no runner pin, so the runner a job gives up can take a newer queued job. The newest finished build never yields.
• 9/9 tests.test_nightly_sign_yield and test_nightly_step_coverage pass locally at this head.

Non-blocking notes:

• listing() fails safe on a total_count mismatch. signing_runner.every_page notes that GitHub's total_count can run ahead of the items for status-filtered run listings (run 40913618519), so this may often read as "unreadable" and sign as before. Paging until an empty page, the way every_page does, would keep the yield working.
• The runs list uses status=in_progress only, so a newer run that already completed is missed. That's the safe direction.
• r2-upload-tests is also red, and the classifier found no base verdict to compare it with. This PR doesn't touch R2, but it's worth a look.

Agent session: agent session

[teamleaderleo]

Review (goal-nightly-30min, independent Claude Opus 5.5): SHIP at a35ac85

The BLOCK is fixed: tests/test_nightly_sign_yield.py is lane = "manual" with a reason, and validate_test_execution_registry.py is valid at this head (633 tests). listing() now reads until an empty page and treats more than MAX_PAGES as doubt (signs), the same as signing_runner.every_page. I checked the edges directly: exactly MAX_PAGES full pages read, MAX_PAGES+1 raises Unreadable, and an empty listing returns []. tests.test_nightly_sign_yield passes 9/9 and test_nightly_step_coverage is OK. The diff since bf25b77 touches only those three files. My earlier logic review still holds.

Agent session: agent session

[teamleaderleo]

Review (codex, router-review): SHIP at a35ac85

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 1. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at a35ac85

Visual verdict: unchanged by this diff

• [P2] [follow-up] [category:correctness] [finding:F1] [outcome:pending] scripts/ci/nightly_sign_yield.py:98 only lists workflow runs with status=in_progress. Active runs waiting on an environment or pending a concurrency slot are reported as waiting/pending and can still contain queued sign jobs, so older builds will miss them and retain the slot. Query all active run statuses (as signing_runner.py does) or add equivalent coverage.

Focused yield, wiring, and nightly routing tests pass.

Follow-ups:

• F1: Cover waiting/pending/requested workflow-run statuses when detecting newer sign jobs.

[teamleaderleo]

Reds at a35ac85, read from every check suite over GraphQL:

• risk evidence CANCELLED (run 41886822901): an earlier attempt that a newer one replaced. 41886891710 passed.
• r2-upload-tests FAILURE (run 41886830953): test_a_slow_large_put_is_bounded_by_idle_gaps_not_total_time ("R2 upload failed: The read operation timed out"). scripts/ci/base_red_excuse.py --run 41886830953: "not excused: no feat-cmux-next run of its workflow has a verdict for it to compare with" (the workflow runs on pull requests only). By hand: this PR doesn't change scripts/ci/upload-r2-object.py or tests/test_r2_upload_requests.py. The same test fails on feat-cmux-next's tip fed3e34 with that tip's two files, and at this head; it is a base red from ci(r2): stream PUT bodies so the 30 s timeout bounds idle gaps, not the whole nightly DMG upload #35337's test. The workflow runs here only because nightly.yml changed. Its failed job is re-running.

Reviews at this head: SHIP from Codex gpt-6.1-sol and from an independent Claude review. Codex's P2 follow-up F1 (read newer runs in waiting, pending and requested status too) is filed as cmuxterm-hq#7091. merge_freeze.py check: exit 0.