Skip to content

Read the org runner roster live; dissolve the pinned-corpus scaffold - #11711

Merged
gunbai-bot[bot] merged 9 commits into
mainfrom
session/swift-moth-559
Sep 19, 2026
Merged

gunbai-bot[bot] merged 9 commits into
mainfrom
session/swift-moth-559

Conversation

@briansrls

@briansrls briansrls commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

What

The runner-capacity planner no longer plans from the pinned 2026-08-06 incident corpus. Both production call sites (gunbc.apply Plan arm and gunbc.runner_capacity_realize runner_capacity_apply_wet_exit) now call gunbc.fleet.organization_runner_roster_read organization_runner_roster_live_read. It returns the existing carrier OrganizationRunnerRosterEvidence. organization_runner_roster_producer_scaffold and its ReadAbsent row are deleted.

  • Route and custody. Reads use the gh CLI with the env-held GH_TOKEN/GUNBC_ORG_ADMIN_TOKEN that gunbc.ci_spec gunbc_ci_org_admin_app_token_prelude mints in-run. This is the same route org_actions_converge already uses for runner groups. No key is copied, no PAT is created, and no token is placed on argv. The organization is derived from the gunbai-ci App row (gunbc.runner_host_deploy github_app_runner_org(gunbc_ci_github_app)), not written as a literal.
  • Pagination. The read follows RFC 8288 Link rel="next" targets verbatim, and every continuation decision (advance, cycle check, page budget, closure) is made by std.page_fold. It is Observed only when three things hold: the listing closed, every page reported the same total_count, and the number of distinct runner ids equals it. Otherwise it is Partial. If a later page fails, the earlier members are kept and the result is Partial. A failed first page, or a missing credential, gives RosterReadUnavailable. It never falls back to the fixture.
    • Stated divergence (§3b): std.page_fold's outcome deliberately carries no items on refusal. The walk keeps its own copy of the pages because the roster carrier has a typed partial arm, and its consumers already decide per question whether a prefix may answer. The fold state remains the only authority on whether another request is owed.
  • Per-ID read. organization_runner_id_observation(org, runner_id) returns RunnerIdPresent / RunnerIdAbsent / RunnerIdUnobserved, each carrying its org. It uses GET /orgs/{org}/actions/runners/{id}. A 404 counts as Absent only after a runners listing of the same org answered 200 under the same token; otherwise it is Unobserved. This was quiet-moth-644's requirement for Retire srv2 as a declared obligation, from an executor that is not the target (L2a-1) #11680: an absence caused by insufficient scope must never discharge a removal obligation.
    • This is a read only. It establishes no DELETE grant and never probes for one.
  • Fleet-converge mode org_runner_roster_observe. It uses the same token prelude, writes a receipt, and exits non-zero unless the roster was complete. The workflow is regenerated via tools.generated_artifact_gate main_wet_one.
  • Upstream model fixes.
    • New extdeps.github.cli_api_include reads the gh api --include framing (status line, Link header, body).
    • GitHubSelfHostedRunnerRegistration.ephemeral and .version are now Optional. GitHub's schema does not require them, and version is string|null.
  • Subject binding (side-chat holds OR-1..3, fixed at 68601cf):
    • A rel="next" target is admitted only as this organization's runners listing: orgs/<login>/actions/runners, or organizations/<id>/actions/runners when the id matches the one read from GET /orgs/{org}. Any other org, repository or resource refuses before fetch.
    • Visibility evidence carries its org and the admitted route. That route needs both GH_TOKEN and GUNBC_ORG_ADMIN_TOKEN, as the org-admin prelude sets them. A 404 is Absent only for that same org.
    • A repeated runner id whose full readings disagree is withheld, and the roster is Partial. Agreeing repeats deduplicate.
    • Witnesses drive the real pump through two-page std.page_fold progressions over scripted responses, including a cross-org continuation and a disagreeing repeat: 20/20 PASS.
  • Fixture. The pinned corpus is renamed incident_2026_08_06_roster_fixture and is used by witnesses only.

Evidence (local claim_batch, this head's tree)

  • organization_runner_roster_read_witness: 13/13 PASS. They cover Link continuation and closure, Optional fields, 403 as a failure, unreadable output, Observed, repeated ids giving Partial, a failed later page keeping members as Partial, a failed first page giving Unavailable, 404 with and without an established listing, and per-ID mismatch.
  • fleet_runner_connectivity_witness: 22/22 PASS.
  • runner_capacity_plan_witness and runner_capacity_realize_witness: all PASS except three that fail identically on unchanged origin/main 7d1d6c5: at_the_committed_width_additions_follow_purpose_not_count, apply_seam_plans_additions_at_the_committed_width_not_count_refusal and apply_seam_preserves_sibling_host_receipt_when_one_refuses. cool-dove-770 is fixing them.
  • Production route executed without a credential: gunbc run --entry dag/gunbc/apply.dag --function apply_wet with workflow=runner-capacity and mode=plan exits 1 with "runner roster read unavailable … GUNBC_ORG_ADMIN_TOKEN is absent … no pinned population is substituted".
  • Credentialed live read, executed. Run: https://github.com/gunb-ai/gunbc/actions/runs/35450620333 (host srv3, head 68601cf, runner_ids=251463,251484,91529,91513), artifact org-runner-roster. Receipt head line:
    organization=gunb-ai pages_read=1 roster=observed count=100 disputed_ids= note=listing closed; distinct runner ids compared against total_count
    • 100 registrations by name prefix: srv1 12, srv2 2, srv3 46, srv4 40. The only srv2-* registrations are 251463 srv2-02-1789774588-3693107 and 251484 srv2-05-1789775039-286764, both offline and not busy.
    • Per-ID probes (real GETs; the 404 path executed live): 251463 present (srv2-02, offline); 251484 present (srv2-05, offline); 91529 absent (404 after a same-org listing answered 200); 91513 absent (same).
    • Caveat: a closed listing of exactly 100 equals max_per_page. Observed rests on two independent API facts, no rel="next" and total_count = 100 = distinct ids, but the coincidence is worth a second run before anything is discharged from it. Per the side-chat hold, this read is evidence and is not yet used to discharge retirement obligations.
  • Discriminating continuation run. Run: https://github.com/gunb-ai/gunbc/actions/runs/35458738938 (srv3, head 62b8dd7, roster_page_size=50, same four probes). Receipt: page_size=50 pages_read=2 roster=observed count=92 disputed_ids= note=listing closed.
    • Page 1: target orgs/gunb-ai/actions/runners?per_page=50, total_count=92, Link: <https://api.github.com/organizations/244123794/actions/runners?per_page=50&page=2>; rel="next", …; rel="last".
    • Page 2: target is that rel="next" URL, used verbatim and admitted because its numeric org id matched the id read from GET /orgs/gunb-ai. total_count=92; Link carries only prev/first, so the listing closed.
    • 92 distinct ids = total_count 92. The population changed from 100 between the two runs, which is expected while registrations churn; totals are not required to match across runs.
    • Probes unchanged: 251463 present, 251484 present, 91529 absent, 91513 absent.
  • Earlier dispatches: srv4 (run 35448023511) failed because gh is absent on that runner (see findings). A srv1 run queued for about 40 min under org-wide load and was cancelled.

Findings

  • The gh CLI is an unmodeled executor dependency. The gh-CLI service family (github.CliOrgRunnerGroups, github.CliOrgSelfHostedRunners, github.CliOrganizations) needs gh on the executing runner, and nothing declares that. gunbc.host_toolchain_components models only rustup components. srv4, a retained runner host, lacks gh (run 35448023511: failed to execute 'gh': No such file or directory); srv1 has /usr/bin/gh. It is not fixed here.
  • A missing executable aborts evaluation with an untyped TypeError. The failure is fail-closed but untyped: the shell operation's typed outputs (success=false plus stderr) never form, so the caller's typed refusal arm (RosterPageFailed, and so Partial or Unavailable) cannot fire and no receipt is written. Location: v1_interpreter.rs dispatch_shell (the failed to execute '{argv[0]}' spawn-error arms). It is an interpreter shell-transport gap and is not fixed here.
    • Being fixed in gunbc#11737 (still-bear-318, base main). A spawn failure becomes extdeps.transports.shell ShellOutcome.ShellSpawnRefused { program, cause } for an operation that declares an outcome: ShellOutcome output. Declared follow-up for this read, triggered by Shell transport: spawn failure is a typed ShellSpawnRefused, not an evaluation-aborting TypeError #11737 landing: add outcome: ShellOutcome to ListOrgRunnersPage, GetOrgRunner and GetOrganization; carry it through RosterTransportResponse; and map ShellSpawnRefused to RosterPageFailed, or to RunnerIdUnobserved for per-ID reads. That is sufficient for a missing gh to yield Unavailable, or Partial with earlier pages kept, plus a written receipt, not an aborted run.
    • Executor prerequisite is in gunbc#11736 (sunny-raven-692, stacked on this branch). It makes gh a declared prerequisite of the executor that runs the read, refused before the first call, and stops loading the fleet SSH key for the two API-only org modes.
  • Mislabel (not fixed here). gunbc.runner_slot_allocation gunbc_runner_slots_per_host maps DerivedRunnerWidthUnresolved { cause } to 0 and drops the cause. So when srv2's commitment is unestablished, runner_capacity_plan refuses with "no committed fleet slot", which conflates unknown with zero. The fix is to return the unresolved result as its own typed case and give the planner a separate refusal for it. It belongs in the slot-allocation/planner lane (cool-dove-770), not in the roster.
  • jit_mint_effect_transport_frontier remains a real, distinct gap. It is a POST whose response is a secret-bearing JIT config, which must reach neither argv, logs nor receipts. An env-authenticated gh api read does not discharge it, and it does not block this read.

🤖 Generated with Claude Code

Brian Searls and others added 2 commits September 19, 2026 14:00
gunbc.apply (Plan) and gunbc.runner_capacity_realize now plan from
gunbc.fleet.organization_runner_roster_read organization_runner_roster_live_read:
GET /orgs/{org}/actions/runners through the gh CLI under the in-run gunbai-ci
installation token (gunbc_ci_org_admin_app_token_prelude), following RFC 8288
Link pages via std.page_fold, accounting by distinct runner id against
total_count. A failed later page keeps earlier members as RosterObservedPartial;
no credential answers RosterReadUnavailable, never the pinned corpus, which
survives only as the dated witness fixture incident_2026_08_06_roster_fixture.

Adds the org-scoped per-ID read (present / absent / unobserved), with 404 read
as absent only after a same-org listing answered 200. Adds fleet-converge mode
org_runner_roster_observe. GitHubSelfHostedRunnerRegistration.ephemeral and
.version become Optional, matching the upstream schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title convergence: live org runner roster producer (dissolve pinned-corpus scaffold) Read the org runner roster live; dissolve the pinned-corpus scaffold Sep 19, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review September 19, 2026 14:07
Brian Searls and others added 2 commits September 19, 2026 14:12
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OR-1: a rel=next target is admitted only as this organization's runners
listing (orgs/<login>/..., or organizations/<id>/... when the id was read
from GET /orgs/{org}); any other org, repo or resource refuses before fetch.
OR-2: visibility evidence carries its org and admitted route (GH_TOKEN from
the org-admin prelude); a 404 is absence only for that same org.
OR-3: repeated runner ids are reconciled over their full reading; a
disagreement withholds the id and makes the roster Partial.
Witnesses now drive the real pump through std.page_fold over scripted
responses. The page bound is re-grounded as a policy budget with the
receipt as its instrument; the observe mode probes captured ids from the
new runner_ids input, consuming the per-ID read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Re review 68461, addressed at 68601cf:

  • Dangling per-ID read. It now has an executing consumer. organization_runner_roster_observe_wet (fleet-converge mode org_runner_roster_observe) calls organization_runner_id_observations for the ids named in the new runner_ids dispatch input (GUNBC_RUNNER_ID_PROBES), and writes a probe id=… org=… present|absent|unobserved line per id into the receipt. That run is what reaches GetOrgRunner on the real 404 path. The retirement closure in Retire srv2 as a declared obligation, from an executor that is not the target (L2a-1) #11680 is its second, named consumer.
  • max_pages: 60 grounded on a transcribed measurement. It is re-grounded as a policy budget: 60 pages of extdeps.github.github max_per_page. The instrument that says whether it is adequate is the observe receipt, which records pages_read and total_count on every run. Exceeding it is loud: PageBudgetExhausted, reported as Partial. The transcribed 522 is removed.

The same head also addresses the side-chat holds OR-1..3 (continuation subject binding, org-scoped visibility, disagreeing repeated ids). Their witnesses drive the real pump through std.page_fold two-page progressions; 20/20 pass locally.

— sent from swift-moth-559

Brian Searls and others added 2 commits September 19, 2026 15:34
Per-ID absence now joins visibility derived from the roster walk the observe
entry already performed, instead of a second first-page listing; the unused
single-id wrapper is deleted; an unparsable runner_ids token refuses the
step rather than being dropped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Re review 68504, addressed at 2988af0:

  • Dangling organization_runner_id_observation. Deleted. The plural organization_runner_id_observations is the one entry, called from organization_runner_roster_observe_with_probes.
  • Duplicated visibility probe. Removed. runner_visibility_from_report derives visibility from the roster walk the observe entry already ran (pages_read > 0 means the first page was delivered). organization_runner_id_observations takes that visibility as a parameter and issues only the per-ID GETs, so an empty runner_ids costs no extra request. Witness: visibility_follows_the_roster_reads_first_page.
  • Silent drop of malformed ids. parse_runner_id_probes returns RunnerIdProbesUnparsable { tokens } for any non-integer or negative token, and the step exits non-zero naming the tokens (OrgRunnerIdProbesUnparsable). Witnesses: a_malformed_runner_id_token_refuses_rather_than_vanishing ("11, 1x2" refuses naming 1x2) and well_formed_runner_id_tokens_parse_in_order.

The roster witness module passes 23/23 locally.

— sent from swift-moth-559

Brian Searls and others added 3 commits September 19, 2026 16:53
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts:
#	dag/gunbc/fleet/fleet_runner_connectivity.dag
#	dag/test/claim/runner/runner_capacity_plan_witness_test.dag
#	dag/test/claim/runner/runner_capacity_realize_witness_test.dag
…es; fail on unobserved probes

OR-2: the route is admitted only when GH_TOKEN and GUNBC_ORG_ADMIN_TOKEN are
both bound and agree; a mismatch or a missing binding refuses before any read.
OR-3: repeated ids are compared field by field (optional presence as presence,
labels through runner_label_sets_equal), not through a joined rendering.
OR-4: the observe command exits non-zero when any requested probe is unobserved;
negative and non-integer ids refuse. Adds a roster_page_size dispatch input
(admitted by std.page_fold admit_page_size) and per-page receipt lines
(target, total_count, Link field, ids).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot
gunbai-bot Bot added this pull request to the merge queue Sep 19, 2026
Merged via the queue into main with commit cf66aa2 Sep 19, 2026
14 of 17 checks passed
@gunbai-bot
gunbai-bot Bot deleted the session/swift-moth-559 branch September 19, 2026 23:56
@briansrls
briansrls restored the session/swift-moth-559 branch September 20, 2026 00:01
gunbai-bot Bot pushed a commit that referenced this pull request Sep 20, 2026
main carries #11711 squashed (cf66aa2), so the roster read and its witness are resolved to
main's content with only this PR's own hunks re-applied: the gunbc.host_cli_dependency import,
org_runner_route_on_executor, and the gh-prerequisite witness rows. fleet-converge.yml was
regenerated rather than hand-merged; the regenerated projection of the merged authorities is
byte-identical to this branch's file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant