[teamleaderleo]

claude-sr2-coord's ask: "glaeda admits the newest nightly run's jobs first on that label, ahead of older runs, without cancelling started jobs" (run 41655282358: 39 min landing to feed, about 6 of it queued on glaeda-nightly-only-m4pro).

Cause

GitHub hands an idle runner the oldest queued job on its label; a glaeda runner cannot pick one. nightly-next is promoted about every 6 minutes and a run takes 40-55, so 5-6 runs share the ten slots of the two nightly-only minis (runs #15787-#15820: every slot that mattered was held by another nightly run whenever #15807's jobs queued). decide's 163 s place-reserved is cmux's portable_job_placement.py re-reading for an idle slot on the same label (REQUIRE_LIVE_CAPACITY), so it is the same occupancy; the light-job move to fleet Linux (nightly-next-freshness lane) removes that wait and is not touched here.

Change (scripts/glaeda-nightly-only-admit only; hook bytes unchanged, so no signing-hook pin)

On attempt 1 of a nightly-next push of nightly.yml, the dedicated admission refuses in seconds a job of an overtaken run, and the runner takes the next queued job: the cmux NEXT feed (appcast-arm64.xml) lists a build above this run's (run id + 2-digit attempt, the same compare as cmux's sign-pick-overtaken.sh). Publication is latest-wins, so the run is certain not to publish; its jobs up to publish-nightly that run here yield. Legacy-identity, deltas and report jobs never yield.

Never: a newer push or a newer build that is only running (it may still fail; and when every run skipped for a newer push, promotions outran every build, cmux #28961), a started job, a re-run attempt, main's nightly (the trusted path), or any doubt (feed unreadable, a non-numeric id): those admit as before.

(The first head also yielded an older run's pre-build jobs while a newer build ran on the same mini; both reviews noted that this is not "certain to fail", so it was removed at 74532ab.)

Measured on #15787-#15820 (183 nightly-only jobs, 10.3 slot-hours): overtaken pre-publish jobs about 0.3-0.5 slot-hours, undercounted because #15799 published after #15805. Most of the rest is older runs' post-build signing, verify and legacy jobs while a newer run builds. Those runs can still publish first, so they keep running.

Known effect: a yielded job fails its run, so report-nightly-failure lists it. A follow-up asks cmux to treat a glaeda yield like its own "replaced" build.

Tests

python3 scripts/test-glaeda-nightly-only-admit.py: 44 passed (new NewestRunFirstTest: scope, overtaken, a newer attempt of the same run, a newer push alone never yields, bad ids admit, never raises, the trusted main path never yields, CLI end to end with a file:// feed).

Agent session: agent session

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Review (codex, router-review): SHIP at e7c524f

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at e7c524f
Visual verdict: unchanged by this diff.

All 44 admission tests passed.

1. [P2] [follow-up] scripts/glaeda-nightly-only-admit:182 — A lost completion record makes a finished build count as running for three hours. The hook’s sampler is best-effort, and this filter ignores subsequent jobs on the same runner that prove the build ended. Reproduced with a build-start record followed by another job starting on that runner: admission still refuses the older build. Revalidate the candidate’s live job identity before refusing; add coverage for missing completion records and runner reuse.

Follow-ups:

• Validate build liveness before yielding to an unmatched start record.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at e7c524f

Visual verdict: unchanged by this diff (no UI).

What I checked: the 44 tests pass. I went through the admission against nightly.yml's needs graph and runs-on routing, and against the hook's job_meta/job_event lines. BEFORE_BUILD_JOBS is exactly build-nightly-app and its ancestors that can land on these runners. Every BEFORE_PUBLISH_JOBS member is upstream of publish-nightly. The legacy-identity chain doesn't depend on any of them succeeding: legacy-identity-target is always() on hosted Ubuntu, and the later legacy jobs gate on !cancelled(). The recovery path can't publish from an overtaken run, because find-nightly-next-recovery.py only takes runs with an id below the current run and a build above the feed's highest, and none exist when this run is already below the feed. So an overtaken run can't publish at all, and refusing its jobs is safe. The newest run is never refused: the feed can't hold a higher build unless a newer run published, and the passed rule needs a newer run's build to be running. A newer push alone never yields, so the #28961 livelock can't come back. Everything else admits as before: a bad id, an unreadable feed or log, a cut-off log tail, telemetry turned off, or any exception. The hook's bytes are unchanged.

Follow-ups:

• [follow-up] Passed rule: if the newer run's build then fails, the older runs that yielded during its build window are gone, though they could still have published. This only covers the newer build's running window, since its completed line re-admits, and nightly-next is compile-gated, so it is rare. Still, it is not "certain to fail". Consider recording yields so a failed newer build can re-run the newest yielded run.
• [follow-up] A yielded job fails its run, so report-nightly-failure lists it (the PR names the cmux-side follow-up).
• [follow-up] NIGHTLY_NEXT_FEED_URL hard-codes nightly.yml's NIGHTLY_NEXT_FEED_BASE. If they drift, the overtaken rule turns itself off without a sign (fail-open, so safe). A test or comment tying the two together would catch it.
• [follow-up] A build whose completed line is lost (the sampler never started) keeps older runs' pre-build jobs yielding on that host for up to NEWER_BUILD_FRESH_S (3 h). Newer runs are unaffected, so there is no livelock.
• [follow-up] Test gaps: the trusted-ref path (--trusted-ref) also reaches newest_first_refusal and has no test, and there is no case for a newer run on attempt 2.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at 74532ab

Checked the BEFORE_PUBLISH_JOBS set against nightly.yml's needs graph. decide is not in the set. legacy-identity-target is always() and does not depend on this run's build: with prepare-nightly-variant refused, CURRENT_BUILD_READY=false and legacy-identity-compare-feeds.sh takes the feed's newest build. pick-nightly-sign-runner already runs for the legacy-only case, and sign-nightly-identity-legacy uses !cancelled(), so legacy publication of the published build still goes ahead. find-nightly-next-recovery recovers only older runs whose build is above the feeds' highest, which an overtaken run can never find, so refusing it loses nothing. The compare matches sign-pick-overtaken.sh: the same feed URL, run_id*100+attempt, strict highest > build, 1-18 digits. Every error path admits: a curl failure or timeout, a feed over 4 MiB (the live feed is about 4 KB), no versions, ids that are not numbers, or an exception. The hook's diff is empty. 44/44 tests pass.

Follow-ups:

• [follow-up] A yielded job fails its run, so report-nightly-failure comments on or reopens the nightly-next failure incident. That usually happens after the newer run's success has closed it, so it shows a false open incident until the next publication. This is already noted in the body. The cmux side should treat a glaeda yield (a failed "Set up runner" with the yield message) like its "replaced" build case.
• [follow-up] owned_pool_rescue watches the dedicated label (nightly_dedicated) and re-runs refused nightly-next jobs unless a newer unfinished run exists or the nightly-next head moved. An overtaken run almost always has a moved head. A newer build published for the same head (a dispatch) would still get a wasted attempt 2 that this admission does not refuse. This is an edge case and should be checked when the cmux follow-up lands.

[teamleaderleo]

Review (codex, router-review): SHIP at 74532ab

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

SHIP at 74532ab
Visual verdict: unchanged by this diff

Checked admission scope, feed comparison, failure handling, and hook integration. All 44 focused tests pass; no actionable findings.

Follow-ups: none