A truncated roster read is a successful one, so absence from a prefix rendered as "the runner is gone"

[briansrls]

The defect

gunbc.fleet_runner_connectivity states, as a note, that the GitHub org runners API "returns every self-hosted registration for the organization in one response." That is false. The endpoint is paginated at 100 rows, and measured 2026-08-27T21:29Z through the gunbai-ci App installation token the org holds 522 registrations — 75 online, 447 offline — so a caller reading one page sees 19% of the roster.

OrganizationRunnerRosterEvidence had two states: RosterObserved and RosterReadUnavailable. A truncated read had no spelling. It succeeds, so it entered as RosterObserved carrying a prefix, and every registration past the cut classified as RunnerRemoteAbsent. At runner_slot_connectivity_verdict_for_incarnation that renders SlotConnectivityOrphaned, and gunbc.runner_capacity_plan connectivity_replace_action maps Orphaned to RunnerIncarnationReplace — a restart.

That is the empty-observation narrow (DESIGN §5): an observation that could not express what it failed to see, rendered as the verdict "these are gone", whose remedy is a restart.

The adjacent case was already defended, which is the receipt that the distinction was understood. roster_unavailable_yields_unknown_not_detached refuses to read a failed read as everything-detached. Truncation slipped past because a truncated read arrives as success.

The repair

• RosterObservedPartial — a variant, not a flag, carrying observed_count and authority_total, so a consumer cannot ask about a slot without the completeness fact in hand.
• organization_runner_roster_from_page derives completeness at the boundary. A caller cannot hand it a prefix and reach RosterObserved; that variant requires observed length to equal the authority's own total. Construction over validation (DESIGN §5) — "I saw all of it" becomes a computed fact about the response rather than a claim about the request.
• The refusal is per-question, not per-roster. A truncated roster still answers exactly about every slot inside the prefix; only the ABSENT arm changes. Refusing the whole read would replace a fail-open with a fail-shut that stops reconciliation on hosts the read covered perfectly well. One shared lookup (registration_verdict_or), two unseen answers — the only thing that differs between arms is what absence means.
• Aggregates refuse rather than report prefix counts. Unlike the per-slot question, every field of FleetAchievedRunnerWidths is an aggregate over the whole roster, so no sub-question is answered correctly by a prefix; a width over a prefix is wrong, not partial, and would read as an outage.
• The false note is deleted, not amended (DESIGN §3 — a corrected sentence beside a carrier that now decides the same question is a second authority for it). Its three surviving facts — org-level read not per-host poll, the App permission, journal mining is not the authority — are enumerated into organization_roster_read_locus_note, which makes no completeness claim. A prose row takes everything in it unless its contents are named first.

Evidence, by execution

Compile on 472cadfcfe: 0 blocking, 435 files emitted.

Five witnesses, values read from the gunbc run refusal (it declines to exit-code a Bool, so the value is in the diagnostic):

witness	baseline	mutated	restored
prefix_below_authority_total_is_partial	true	—	—
page_equal_to_authority_total_is_complete	true	—	—
truncated_roster_never_orphans_an_unseen_slot	true	false	true
complete_roster_still_orphans_an_unseen_slot	true	true	true
truncated_roster_still_answers_slots_inside_the_prefix	true	—	—

The mutation is the evidence. Flipping the partial arm's unseen answer back to Orphaned flips the RED to false while the control holds at true — so the RED measures this repair rather than coexisting with it, and is not tracking a globally disabled Orphaned. The carrier was then restored byte-identically (cmp) and both re-run.

The complete-roster control is doing the heavier work: Orphaned must SURVIVE the repair, because a slot absent from a roster the authority itself calls complete is genuinely gone. The minimum fix is not the smallest thing that stops the false restart — it must preserve the required refusal.

Both fixtures derive authority_total from the specimen list, so neither can silently change meaning when a registration row is added. An earlier revision asserted seen == 16, a literal transcribed from the tree under test — the oracle DESIGN §5 rejects.

Rung

Mechanically preventable for the model. The invalid state remains writable — an author can still construct RosterObserved directly with a prefix — and safety depends on the witness executing and staying enrolled.

NOT structurally guaranteed for the fleet. The running runner-liveness-reconcile.sh still requests ?per_page=100 with no pagination and still acts on 19% of the roster. This PR does not change one byte of what executes on any host. Claiming otherwise would be the rung inflation §4b calls worse than sitting low.

Next-rung trigger: a render route from this authority to the host script, analogous to what gunbc.runner_unit_file did for actions-runner@.service in #9401. That route does not exist, and building it is a replacement migration rather than a defect fix. The only alternatives available today — hand-editing a host, or committing a second hand-authored shell beside the model — are both forbidden (DESIGN §6, out-of-band actuation and parallel authority). Escalated to the operator; not resolved here.

What this PR does NOT claim

It fixes no observed cancellation. It repairs a false belief about the roster. That is the whole claim.

The credential-deletion mechanism that motivated the investigation — an orphaned Runner.Listener, surviving KillMode=process, deleting a live incarnation's .credentials mid-job — is established at n=2 by pid-level evidence, and does not explain the 385 completed with result: Canceled events on srv4 since 2026-08-25.

	srv4-18, job 98644348200	srv4-04, run 33130742525
reconciler restart	19:24:04 WEDGED → restarted	00:35:24 WEDGED → restarted
orphan survives	pids 4024076 / 4024080	pids 1425187 / 1425194
job pid	4054700 (19:25:48)	1469326 (00:45:47)
credentials deleted by	4024080 at 19:29:24	1425194 at 00:56:52
noticed at refresh	20:14:55 (45.5 min later)	01:26:28 (29.6 min later)
cancelled	20:15:11	01:26:46

Both are OTHER-PID-DURING with the reconciler's own WEDGED … restarted line inside the window.

The commit message on this branch says the deletion is latent until "the ~50 min OAuth refresh". That is wrong, and the second specimen is what falsified it. The latency is the time to the next refresh — anywhere in (0, ~50] depending on where in the cycle the deletion lands — which is why the srv4-04 floor died at 41 minutes rather than past 50. The commit is left unamended because history is immutable; this paragraph supersedes it.

The corrected predicate contains no duration term: a job dies iff an orphan deletes its credentials during the job and a token refresh occurs after that deletion and before the job ends. Duration is a proxy only because a longer job is likelier to contain a refresh. That agrees with the independent finding from the API side — incarnation-epoch decoding across 58 floors showed successful non-srv4 floors on incarnations 57–115 minutes old, so no age clock kills jobs — two instruments, same conclusion.

A counterexample still sits inside our own sample and is stated rather than closed: srv4-08's 54.0-minute floor cancelled at 20:31:06 with no orphan removal and no rsaparams event — every credentials removal same-pid-after-completion. Cause unidentified.

Duration and job type are confounded in every specimen produced during this investigation. Both candidate controls (srv4-11/srv4-18, srv4-08 back-to-back) vary job type along with duration. This is a limitation of the evidence base, not a caveat on a conclusion.

Also refuted during the work, from inside our own data: srv4-01 read registration=absent on every reconciler tick while running a 61.9-minute floor to a real FloorRefused verdict. Invisibility makes a slot eligible for a false wedge; it does not kill it. The kill additionally requires an actual restart landing inside a running job.

Two findings that outlast this defect

1. A file asserting it is version-controlled when no repository holds it. /opt/actions-runner/runner-liveness-reconcile.sh opens with "Load-bearing: canonical, version-controlled copy... edit here (not the installed copy) and re-run the installer." It is in no repository. git log --all for liveness in gunb-ai/ctrl is empty and install-actions-runner.sh never references it. The claim fails toward "someone else already owns this", which is why nobody found it.

2. The census built to catch exactly this class was structurally blind to it — the more valuable of the two, because it generalises. gunbc.ctrl_host_provisioning_census exists to enumerate every fact the ctrl host-provisioning surface decides, one row each, owned or unowned-with-reason. It has no row for this script, because its denominator is the ctrl repository and the file is not in it. Anything provisioned on the hosts and absent from the repo is invisible the same way.

Enumerated on srv2 — host-present, repo-absent:

runner-liveness-reconcile.sh
runner-liveness-reconcile.service
runner-liveness-reconcile.timer
ctrl-fleet-converge.service
ctrl-fleet-converge.timer

The last two are the unit that has been panicking on every 15-minute tick (workspace_root: / is not inside a git checkout, v1_compiler/cli_run.rs), converging nothing for as long as the journal reaches back.

A second member, different mechanism, same blindness. Measured on srv1 2026-08-27 and reported in #9506: /opt/gunbc/gunbc carries 5650 divergent paths — 1964 tracked-and-absent, 1731 unknown to HEAD, 1500 differing from HEAD in the index and from the index in the worktree. Both status columns wrong, from one execution, no concurrency. Three transports over one repository and nothing makes them agree.

The pair is what makes this a class rather than an anecdote, because the two members are invisible for different reasons:

• Mine is a FILE the census could not see, because its denominator is the ctrl repository and the file is not in it.
• Theirs is CONTENT the census could not see, because the transport was never modelled as one thing.

Both are host state that no repository authority produced.

A third finding, about how the first two survived: a stale claim repeated reads as corroborated

#7940 ("PR B: runner connectivity recovery — replace/repair controller (model + witnesses)") merged 2026-08-07. gunbc.runner_connectivity_recovery ports the ctrl watchdog — its own note says so — and contains no HostEffect, no realize, no emit, no render. Model and witnesses, no production caller. Meanwhile the script it "subsumes" has run unsupervised ever since.

The subsumption is asserted in the present tense in two carriers:

gunbc.runner_lifecycle              "#7940 subsumes it"
gunbc.runner_capacity_operator_access   "#7940 subsumes runner-liveness-reconcile.timer"

The second occurrence does more damage than the first. A stale claim repeated in two carriers reads as corroborated: a reader greps, finds two independent-looking assertions, and stops. That is the sentence that stopped anyone looking at the reconciler, doubled.

The contrast that makes this actionable rather than a complaint: gunbc.live_deploy.repository_convergence (#9506, merged same window) is in the identical state — model, witnesses, no production caller — and declares it in the carrier itself: "NOTHING HERE IS BOUND. No production caller reaches this module." It cannot be read as done. Same state, opposite legibility, and the legible one costs nothing later.

Neither finding is repaired here. All are named so they rank.