Skip to content

Fleet converge gaps: ephemeral registrations retire, teardown grant enrolls the slot grammar, diverged is not unrelated - #10485

Merged
briansrls merged 3 commits into
mainfrom
fleet-converge-gaps
Sep 5, 2026
Merged

briansrls merged 3 commits into
mainfrom
fleet-converge-gaps

Conversation

@briansrls

@briansrls briansrls commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Four climbs, each found by running fleet converge against the real fleet on 2026-09-04 and each fixed at its authority.

1. A dead ephemeral registration is a retired incarnation

srv2-11 and srv2-12 sat refused for months: a .runner file was present, so registered: Bool was true. The file says "Ephemeral":"True", and GitHub unregisters an ephemeral runner after one job, so there was nothing left to config.sh remove and no credential was needed to know it.

  • extdeps.github.actions_runner: actions_runner_registration_decode reads the file's Ephemeral member (cited; closed set True/False, anything else unreadable).
  • gunbc.runner_slot_provision: RunnerSlotRegistration = SlotUnregistered | SlotRegisteredPersistent | SlotRegisteredEphemeral | SlotRegistrationUnreadable, replacing the Bool. Retired iff in-grammar, unregistered-or-ephemeral, inactive, disabled. Persistent and unreadable still refuse (persistent needs an org-scoped observation this executor does not hold).
  • The plan-time observer reads the file's bytes only when it is present.

2. The teardown grant enrolls the slot grammar, not the current width

The sudoers roster named rm -rf only for desired slot dirs, so it authorized exactly the removals that never happen and refused every retired tree on a narrowed host. runner_slot_teardown_grant_index_bound = 64: the removal grant names every dir the host grammar renders up to the bound, still one exact path per line, no wildcard. Witness pins the bound above every committed width and checks srv2-64 is named and srv2-65 is not. Regenerated provisioning/srv*/gunbc-ghrunner.sudoers (4 files) through the generated-artifact gate.

3. Diverged is not unrelated

The srv1 deploy refused with "share no ancestry" when the running release was a sibling of the candidate with a merge base two commits back. DivergedHistories is now its own arm of DeployRevisionRelation with its own cause; UnrelatedHistories stays for merge-base exit 1; the two-probe fold declares it cannot distinguish them rather than picking one. AdvanceHistoriesDiverged added on the fleet-desired side.

4. Refusals name their subject

  • The plan artifact write refusal lists which path refused and why (the srv2 run refused bare on /tmp/fleet-converge-plan owned by another principal from a local plan run). The shared literal dir is the underlying defect and is declared, not solved.
  • The fleet-converge job roster witness counts the four jobs main actually has (was red on main).
  • witness_population_tracks_the_slot_roster derives the allocation-store term it had omitted (was red on main).

5. The org-admin credential is minted in-run; no human token

The interim design (operator mints a fine-grained PAT, pastes it into GUNBC_ORG_ADMIN_TOKEN) was never taken and is superseded. The gunbai-ci GitHub App is installed on the organization, its private key is in Secret Manager behind the same WIF read as the fleet key, and the runner installer already mints registration tokens with it. gunbc_ci_org_admin_app_token_prelude fetches the key in-run, signs a ten-minute RS256 JWT with openssl, exchanges it for a one-hour installation token at the cited endpoint, classifies anything but 201 as OrgAdminInstallationTokenRefused, wipes the key, and exports the token into that step's environment only. The workflow step no longer references any repository secret. Witness asserts the endpoint, the secret version path, the signing, the refusal name, and the absence of secrets.GUNBC_ORG_ADMIN_TOKEN in the emitted YAML. A live org_actions_observe run from this branch is the executing proof; its result is recorded in the PR comments.

Evidence

Witness rows PASS: runner_slot_provision 8/8 (incl. ephemeral retired, active-ephemeral and unreadable refuse, decode with RED controls), executor_privileged_operation 2/2, deploy_revision 10/10, target_decision 3/3, fleet_main_revision 3/3, workflow_dispatch_input roster row. fleet_converge_plan_cli.dag closure typechecks with 0 blocking. The wet matrix file is a declared read-live-tree class excluded from the hermetic route; its sibling row is renamed to is_diverged consistent with the model.

Not in this PR

  • Persistent-registration deregistration still needs an org-scoped removal token (GUNBC_ORG_ADMIN_TOKEN is not set on the repo).
  • The Spark observe mode still cannot ssh from a runner.
  • Getting srv1 onto a main revision: it runs a local cherry-pick tip that main does not descend from, so the deploy correctly refuses as diverged. The honest routes are an idle-target redeploy (stop the unit, deploy main) or a patch-equivalence arm; neither is taken here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci

…nt enrolls the slot grammar, diverged is not unrelated, and refusals name their subject

Four climbs measured against today's fleet, each at its authority:

- Slot provenance carries a registration KIND, not a Bool. srv2-11 and srv2-12 sat refused for
  months because a `.runner` file was present; the file says Ephemeral=True and GitHub unregisters
  an ephemeral runner after one job (extdeps.github.actions_runner
  actions_runner_registration_decode, cited), so a dead ephemeral registration is a retired
  incarnation and needs no org credential to retire. A persistent registration still refuses.
- The retired-tree removal grant enrolls the host's slot grammar up to a declared index bound (64)
  rather than the current desired width, still one exact path per sudoers line with no wildcard, so
  a narrowed host no longer refuses at apply the removals a width change created. The bound is
  pinned above every committed width by witness.
- DivergedHistories is its own arm of DeployRevisionRelation. The srv1 deploy refused with "share no
  ancestry" for a sibling with a merge base two commits back; the neither-ancestor case is now named
  as diverged, the no-common-ancestor case stays unrelated, and the two-probe fold declares it cannot
  tell them apart instead of picking one.
- The plan artifact write refusal names which path refused and why. The srv2 run refused bare on a
  /tmp/fleet-converge-plan owned by another principal from a local run; the shared literal dir is
  the underlying defect and is declared here, not solved.
- The fleet-converge job roster witness counts the four jobs main actually has.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-04T23:12:30.639987Z 6a15db2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a15db24b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

fn runner_slot_registration_is_retired(r: RunnerSlotRegistration) -> Bool {
match r {
SlotUnregistered => true
SlotRegisteredEphemeral => true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Require evidence that an ephemeral registration has expired

An Ephemeral flag only means GitHub unregisters the runner after it completes a job; it does not prove that a job has already completed. If an out-of-roster unit is disabled and inactive because it stopped after registration but before accepting/completing its first job, this arm classifies its still-live provider registration as retired, and reconciliation emits rm -rf for the slot instead of refusing like it does for persistent registrations. Preserve the fail-closed behavior until there is evidence that the registration completed a job, expired, or is absent from GitHub.

Useful? React with 👍 / 👎.

gunbc-ci-auto-heal and others added 2 commits September 5, 2026 03:38
… no human token, no repository secret

The interim design asked the operator to mint a fine-grained PAT and paste it into
GUNBC_ORG_ADMIN_TOKEN. It was never taken and was never needed: the gunbai-ci GitHub App is
installed on the organization, its private key is in Secret Manager behind the same WIF read the
fleet key uses, and the runner installer already mints registration tokens with it.

gunbc.ci_spec gunbc_ci_org_admin_app_token_prelude follows the fleet-key lifecycle (WIF token by
env, the SecretRef's own access URL, 0600 under RUNNER_TEMP, trap before the key touches disk),
signs a ten-minute RS256 JWT with openssl, exchanges it for a one-hour installation token at the
cited endpoint (extdeps.github.org_admin_auth github_app_installation_access_token_url), classifies
anything but 201 as OrgAdminInstallationTokenRefused with the status line, wipes the key, and
exports the token into that step's environment only. The workflow step drops its
secrets.GUNBC_ORG_ADMIN_TOKEN rows and takes the WIF access token instead. The acquisition plan's
interim section is marked superseded; the witness asserts the endpoint, the secret version path,
the signing, the refusal name, and the absence of any repository secret reference.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…or binding is a converge the control plane runs

The first live org_actions_observe run refused with a JSON decode traceback: Secret Manager answered
403 for ci-github-app-private-key to the fleet-cloud-convergence principal, and `curl -sSf | python3`
collapsed that into "Expecting value" -- the exact status-collapse the administrator prelude's note
records. The read now captures the HTTP code, refuses OrgAdminAppKeyUnreadable naming the version
resource, the principal, the 403 ambiguity, and the remedy, and decodes only a 200 body.

The remedy is modeled rather than a hand gcloud line: gunbc.spark.secret_access_ensure is generalized
over its target secret (secret_access_ensure_for; the spark entry is one caller), and
gunbc.fleet.org_actions_converge gains org_admin_app_key_access_converge_with_supplied_token, which
reconciles roles/secretmanager.secretAccessor on the App key for the workload principal using a
control-plane token from GUNBC_GCP_ACCESS_TOKEN_FILE. The 14 spark access rows still pass against
the generalized ensure; the dispatch witness asserts the classified read.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
@briansrls

Copy link
Copy Markdown
Contributor Author

Live proof of the in-run App-minted credential (run 33943347397, dispatched from this branch after the modeled accessor converge bound roles/secretmanager.secretAccessor on ci-github-app-private-key):

  • Secret Manager key read: 200
  • org-admin: installation token minted in-run for app gunbai-ci installation 104134109
  • GET /orgs/gunb-ai/actions/runner-groups and the per-group repositories read both succeeded with the minted token; the capability receipt was written.
  • The step then exited 1 on OrgActionsSettingsDiverged access=2 public_exposure=0: a real drift between the modeled runner-group access policy and the org, which is exactly what observe mode exists to report. The credential path is closed; the drift is the next converge.

The first run (33942418536) refused on the key read with a collapsed JSON traceback; the classified OrgAdminAppKeyUnreadable read in f740b47 is what replaced it.

@briansrls
briansrls merged commit a607d79 into main Sep 5, 2026
5 of 7 checks passed
@briansrls
briansrls deleted the fleet-converge-gaps branch September 5, 2026 04:13
gunbai-bot Bot pushed a commit that referenced this pull request Sep 5, 2026
fleet-converge-gaps (#10485) landed on main, so the three commits this branch was stacked on are
now upstream and the diff reduces to the widening this lane owns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UBLsJSnmB8ZdDTh1RRVTe7

# Conflicts:
#	dag/gunbc/ci/ci_spec.dag
#	dag/gunbc/fleet/org_actions_converge.dag
#	dag/gunbc/spark/secret_access_ensure.dag
gunbai-bot Bot pushed a commit that referenced this pull request Oct 2, 2026
…ired 2026-09-05

standing: Standing -> Retired { trigger_fired: "2026-09-05" as NonEmptyStr };
the import narrows with it. The trigger text named the restoration route
(tools.emission_entry_instrument measure_entry_emission) and #10485 landed
exactly that route, so the row's temporary state is over; the row stays as
historical evidence of the lost per-entry route, which is what Retired
means here. Consumers key on rung identity, not on the Standing variant,
so no consumer changes.

Projection re-rendered by docs_projection_gate regen: the row's heading
gains the ' · RETIRED' suffix, the body gains the '**RETIRED -- TRIGGER
FIRED.** 2026-09-05' line, and the trigger-fired sentence now says which
commit landed the route instead of asserting its own truth.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant