Skip to content

CONVERGE-0 acceptance: a merged main revision becomes observed fleet state, including offline catch-up — open until the receipt set exists - #8083

Merged
briansrls merged 62 commits into
mainfrom
session/wise-moth-430
Aug 10, 2026
Merged

briansrls merged 62 commits into
mainfrom
session/wise-moth-430

Conversation

@briansrls

@briansrls briansrls commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Immutable contract

Starting: implementation merged (#8045, 147ebbc283); acceptance open; live ending standing unobserved.

Ending: main advances → every applicable host observes it → hosts reconcile → receipts collected → fleet verdict produced → one host absent during an advance returns and converges with no new commit.

Receipt set: desired revision · timer/service standing per host · host receipt per host · lifecycle observation per required cell · fleet fold · offline catch-up receipt.

Displacement, honestly: acceptance receipt 0/1. It does not advance until #8085's merge run actually executes deploy/advance/collect AND refs/fleet/desired == #8085 merge revision, the collector's per-host reads, and the fold are observed.

Current standing — the observation ladder (updated 2026-08-09, run 31333011288)

  1. Name resolution — DONE (modeled Spark endpoints).
  2. Host identity verification — ScannedFirstContact: six host keys scanned + enrolled + materialized in-run. Honest name: this proves the key seen at scan equals the key later accepted, NOT that the first key belonged to the intended physical host (no BMC/console cross-check).
  3. Authentication — DONE via WIF → SM versions/1 pinned → in-run ssh-agent.
  4. Command reach — CROSSED on srv1 and srv2 (gunbc-fleet principal, ProbeReachable, identity Converged, run 31333011288).
  5. Per-host standing today:
    • srv1/srv2: command reach established.
    • srv3/srv4: automation principal + key standing ABSENT (gunbc-fleet not created; typed auth refusals).
    • srv5/srv6: principal standing ABSENT (Spark accounts unverified).
  6. Timer bootstrap — srv2 installed+enabled (run 31335514960); a host-targeting defect remains (dispatch does not select a host — see the repair batch).
  7. Main actuator run — PENDING (Repair: push-to-main deploy/advance/collect silently skipped on every main push (implicit success() cascade) — status guard + RED control #8085 merge run ci job in progress); items 1/2 unobserved until it executes.

Carried in this PR

In-flight repair batch (operator scan of #8083, 7 items): principal-target refusal (no SSH on unobserved principal), principal verification (id -un), failure-safe credential trap, private-key-vs-modeled-fingerprint check, host-targeted bootstrap roster, stronger scan/enroll readbacks + ephemeral known_hosts, and in-memory receipt fold (no static transcription as continuing authority).

🤖 Generated with Claude Code

https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y

Brian Searls and others added 30 commits August 8, 2026 04:08
…onverge circuit

First subject of the fleet-convergence vertical: the desired per-slot
TasksMax=512 (gunbc.runner_slot_allocation, operator ruling 2026-08-05)
becomes an appliable, readback-proven converge knob instead of a row the
fleet cannot act on. Live pre-state on srv1/srv2 is the 2026-06-21
tourniquet TasksMax=16384 (compute_envelope_model), which now reads
Drifted, never Converged.

- extdeps.systemd: TasksMax/TasksCurrent properties + SystemdCgroupTaskLimit
  parse (count-or-infinity), mirroring the memory limit parse with a
  declared consolidation dissolve-on.
- gunbc.host_converge: per_slot_tasks_max knob on the property-generic
  PerSlotMemoryCap target (over-narrow name noted with dissolve-on to the
  P3 axis framework); desired threaded from runner_slot_allocation through
  RunnerHostDeployment.
- gunbc.host_axis_caps: 50-fleet-tasks.conf joins the managed drop-in set
  (Owned, not foreign).
- gunbc.host_converge_slice1: full tasks circuit — live TasksMax read ->
  verdict -> pre-apply TasksCurrent guard (starving clone() with EAGAIN is
  the guarded harm) -> set-property -> independent read-back ->
  reapply-noop; per-unit outcomes beside the memory knob on srv1/srv3;
  refusals typed and located throughout.
- fleet_converge_cli: srv1 slice1 route handles the tasks arm; whole-loop
  refuses if the tasks receipt fails its validity gate.
- Witnesses: knob shape on srv1+srv3, tourniquet-16384/infinity RED
  (Drifted), 512 converges + off-by-one drifts, empty/junk wire is Absent
  (fail-closed), legacy-arm routing, and a validity-gate RED where flags
  claim success but the read still shows 16384.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…conflict)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…84; 512 is the RED

Operator ruling 2026-08-08: TasksMax=512 was the diagnosed EAGAIN outage
value, not the repair — the earlier revision inverted the incident
diagnosis. Corrections:

- runner_slot_allocation: tasks_max 512 -> 16384 at the single authority,
  note updated with the ruling; any lower derived envelope value must be
  separately accepted and never below live task demand.
- Witnesses flipped: 512 (and infinity, and off-by-one 16383) read
  Drifted, never Converged; the validity-gate RED reads 512.
- Guard boundary tightened to >= (equality is zero headroom) via an
  extracted pure predicate, with an equality-boundary witness (manager
  review finding 2).
- Reapply-noop guard bypass justified in-code by physics (finding 1).
- No live apply executed at any point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…able TasksConverged CLI arm

Manager review on #8045: the ConvergeTarget comment described the tasks
knob as TasksMax=512 — this PR's own prose citing the outage value as the
knob's value; the desired is owned by gunbc.runner_slot_allocation and
the literal is dropped. The fleet_converge_cli TasksConverged arm is
annotated unreachable-by-construction so no new caller wires through it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…o real producers, and the fleet-atomic-convergence fold

Item 1 (B): host_phase_observation's hard-coded single-cell if-chain
generalizes into ONE canonical observe_host_phase(subject, host,
phase_label) seam — never a per-program observer beside it. Two real
producers land as pure deductions over committed probe receipts
(gunbc.host_reach_identity_probe), the same shape as the srv3
lease-table producer, so the matrix stays hermetic by construction:

- ONE wet probe (authenticated short-hostname read over the management
  transport) feeds BOTH P0 cells: reach-secrets-network and
  host-identity-converge.
- Four distinct typed outcomes by operator constraint: Reachable /
  TransportUnavailable / AuthenticationRefused / HostKeyMismatch —
  auth-refused and transport-unavailable stay distinct arms (different
  remedies) even where both project Inaccessible; host-key mismatch is a
  live class (srv5/srv6 shipped identical factory host keys) and maps to
  Conflict so a wrong machine's hostname never launders into an identity
  observation.
- Wire classification is fail-closed: unclassifiable failure is
  TransportUnavailable with cause preserved, never fabricated Reachable.
- Receipts registry is honestly EMPTY: every host reads UnknownRefused
  on both cells until a wet probe executes; coverage grows only by
  executed observation, never by wiring. Password auth on srv5/srv6
  stays enabled until a Reachable receipt exists (ordering law recorded
  at the outcome type).

Item 2: gunbc.fleet_convergence_verdict — the fleet-atomic fold joining
knob-grain ConvergeCliReceipts with the lifecycle cells. Laws, each with
an executing witness: a required unobserved cell PREVENTS fleet
convergence (never defaults green); knob-converged receipts alone cannot
converge the fleet (RED); an empty enrollment refuses; unobserved /
unreachable / drifted / conflict stay distinct standings with distinct
remedies; the not-converged arm carries full standings so the verdict IS
the work list. The named current defect is now executable: the live
fleet reads NOT converged with a positive unobserved-cell count (one
real producer cell fleet-wide).

Witnesses: 7 probe + 5 fold, green by execution; host_phase_status's 22
witnesses green under the new seam signature.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…srv1-at-512 control notes

Item 3 (gunbc.fleet_main_revision): refs/fleet/desired as the single desired-state
ref; FleetDesiredRevision (Admitted|Unobserved), admit_fleet_desired_advance is a
CAS carrying the expected prior and REFUSES without floor-green; an unobserved ref
yields NO converge target, never the invoking tree. 4 witnesses green incl. the
floor-not-green refusal RED.

Item 4 slice 1 (gunbc.fleet_converge_timer): single authority for the per-host
ctrl-fleet-converge.service/.timer names, cadence (*:0/15, RandomizedDelaySec=90),
and unit CONTENT — Option A (OnCalendar + Persistent=true, catch-up for offline
hosts); the service converges SELF only via the pinned binary. 4 witnesses green.

Stale-note recut (review finding, flagged twice): runner_lifecycle
runner_lifecycle_memory_cap_apply_note and runner_capacity_operator_access
srv1_already_converged_caps_control_note recorded srv1's hand-converged
TasksMax=512 as part of the acceptance-row-6 no-op control; per the 2026-08-08
ruling 512 is the diagnosed outage value (desired 16384), so both notes now scope
the zero-effects premise to the MEMORY axes and record srv1's tasks axis as live
drift the per_slot_tasks_max knob corrects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ing-built unit INI scaffold)

per_runner_tasks_max gains the 🟡 marker naming the std.measure TaskCount
dissolution (bare-Int pids count beside ByteSize siblings, retiring the
to_string re-serialization); fleet_converge_timer's concat-built unit text is
marked as a tests-only scaffold dissolving on systemd-unit grammar-rows emit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
… row for string-built unit text

fleet_converge_timer_randomized_delay rides std.measure Second (rendered via
second_count at the INI seam); the prose-only string-built-medium marker becomes
a Disposition = Scaffold row binding the live_deploy emit-shell dissolution
trigger, matching the fleet_show_effective_read peer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…-item grain

The 4c annotation channel admits only standalone leading // blocks attached to
module-scope declarations; the marker sat inside the RunnerHostDeployment body
and refused. Moved above the type declaration; host_converge_delta witnesses
green by execution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…r ruling 2026-08-09)

16384 is annotated as an emergency parity rollback only — not a derived value,
not the accepted model home. The named follow-up model distinguishes runner-slot
task budget, shared compile-pool task budget (sccache children live in the
server's cgroup), and host-parent reserve, each derived from one measured
envelope with sum(slots)+pool+reserve <= parent; dissolve-on that model landing.
Not built in this PR per the ruling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…y, srv1-pinned)

Approved by manager (msg_03ec1f78) to produce the operator-ruled blocking
pre-apply receipt without consuming the supervised window. New module
gunbc.srv1_tasks_preapply_observation: strictly read-only wet entry returning
the typed Srv1TasksPreApplyReceipt — per slot unit: effective TasksMax +
TasksCurrent (typed systemctl show seam), cgroupfs pids.max/pids.current at the
ControlGroup path SYSTEMD reports (never a hand-escaped slice path — the decoy
class), the ancestor slice pids.max, plus the observer's own /proc/self/cgroup
membership so the receipt shows the read ran inside the observed universe.
Refusal arms are typed per unit; empty enumeration refuses rather than
reporting a vacuous receipt. extdeps.systemd gains the ControlGroup property
arm. The job is authored in the ci_workflow typed builders, gated
`github.event_name == 'workflow_dispatch'` at the job (never rides PR/push),
pinned to the srv1 runner spec, needs only the release-bins build; ci.yml
regenerated via the artifact gate. Hermetic witnesses green for the pure
ancestor-path logic (parent-of-reported-ControlGroup + degenerate boundary);
ci_workflow witness green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
First dispatch (run 31288091024) captured the full receipt but exited 2 through
the NotProcessExit refusal arm. The wet entry now writes the typed receipt to
target/srv1-tasks-preapply-receipt.txt (Filesystem.Write, workspace-only) and
derives its exit from the receipt's own arm — Observed -> ExitSuccess,
Refused -> exit_failure(cause) — never from transport exit codes; the job step
cats the file so the receipt also lands in the log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ll; feature-tag the two task-count unit gates

gunbc_ci_srv1_tasks_observation_invoke now carries the sibling pattern exactly
(Scaffold disposition row binding the invoke decl + dissolution-trigger string
emitted as the script's # 🟡 header); the per_runner_tasks_max marker and the
SystemdCgroupTaskLimit consolidation note both gain the
feature:task-count-measure-carrier tag naming the std.measure TaskCount
dissolution. ci.yml regenerated; ci_workflow witness green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…cepted 16384)

The two acceptance-row-6 notes still framed srv1's tasks axis as live drift at
512 pending correction; the executed, operator-accepted pre-apply receipt (run
31288091024) reads all five slots at effective 16384 with matching cgroup
pids.max, so both notes now record the observation-decided standing — the
no-op control is exact on every converged axis, and no state-changing tasks
apply is pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…in→desired-ref advancement

CI red root cause: cell_standing_is_converged and host_knob_standing_is_converged
matched closed coproducts with top-level wildcards — the exact class the
non_fold_residue receipt refuses; both now enumerate their arms (construction
over wildcard) and the receipt is green by execution.

Loop leg: --host self — converge_cli_resolve_self resolves the literal "self"
to the kernel-reported hostname at the cited procfs authority
(ProcSysKernelHostname arm added), any other value routes unchanged, and an
unreadable hostname falls through to the UnknownHost typed refusal, never a
fabricated identity.

Loop leg: main→desired-ref advancement — advance_fleet_desired job (push-to-main
only, needs [ci] as the floor-green admission, contents:write) performs the CAS
advance of refs/fleet/desired via --force-with-lease naming the exact observed
prior (fleet_main_revision's from side); creation is the from=none arm. Script
carries the scaffold row + emitted 🟡 header; ci.yml regenerated.

Witnesses green: fleet_converge_cli routing, fleet_convergence_verdict (5),
non_fold_residue receipt, ci_workflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…m a Class B refusal); flat_map for the per-unit outcome accumulator

host_reach_identity_probe and fleet_converge_cli now import std.algebra trim
explicitly. Review 50742: the slice1 per-unit outcome fold was a copied
accumulator (concat(acc, [..]) per unit, O(n^2)); §6 bare-minimum-cost fixes a
proven cost shape regardless of n — now units.flat_map. Slice1 witnesses green;
generated-artifact regen green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ication on the self route

Leg (c): FleetConvergeTimerUnit joins live_deploy membership on the belt-timer
precedent exactly — per-instance names derived from the deployment stem
(unconditional membership, twin-safe by path construction), one member owning
.timer + .service, apply writes both units (service: oneshot root ExecStart
'<deployed binary> converge --host self'; timer: OnCalendar/Persistent=true/
jitter derived from the gunbc.fleet_converge_timer authority rows) with
daemon-reload/enable/restart, teardown removes both files, retirement stops the
runtime. Persistent=true on the calendar timer is the offline-host catch-up
property.

Leg (d): on the SELF route the converge CLI publishes its receipt line to
/opt/gunbc/receipts/<host>-converge-receipt.txt before printing; publication
failure fails closed (converged=false, typed reason) because the loop's product
is the published receipt. Non-self supervised routes unchanged. The deploy arm
installs the receipts dir from the same authority row.

Witnesses green by execution: the two new member witnesses (install content
incl. catch-up + receipts dir; retract removes both files), the twin-safety
controls, and the full 45-witness live_deploy emit corpus.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…and threads without fabrication

Review 50695: the wire classifier stubbed expected_fingerprint as "" — conflating
"enrolled expectation contradicted" (wrong-machine vs rotated-key, comparable)
with "transport flagged verification, nothing enrolled to compare". The field is
now String? (Absent = no enrollment), the wire classifier alone produces only
Absent, and classify_reach_probe_wire_with_expectation carries an enrolled
fingerprint into the same arm without re-parsing the wire. New discriminating
witness; all 8 probe witnesses green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
… fleet reach probe job (loop leg e)

CI red on 1178e7d was the mismatch-arm annotation sitting inside the coproduct
body — hoisted above the type declaration (same 4c class as the earlier
ci_runner_placement red).

Leg (e): probe_fleet_reach_wet — ONE wet pass over srv1-srv6 through the typed
SSH PROBE seam (typed_argv_probe_over_ssh, expect: OutcomeIsData), so a failed
leg is DATA classified into its typed arm (Reachable / TransportUnavailable /
AuthenticationRefused / HostKeyMismatch with Absent expectation — no enrollment
registry exists yet), never a step failure and never an ExpectSuccess annotation
standing where a check was owed. Receipt written as a typed value + step cat;
only a receipt-write failure fails the run. Exposed as workflow_dispatch-only
job probe_fleet_reach on the srv1 runner (the probe runs FROM inside the fleet),
scaffold row + emitted 🟡 header per the transport discipline; ci.yml
regenerated.

Also: reapply-noop comment wording per review 50754 (non-atomic read/write
stated; the class cannot be CAUSED by re-applying the effective value,
regardless of concurrent spawns).

Witnesses green: 8/8 probe (incl. expectation threading), ci_workflow golden.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…e sites

host_reach_identity_probe's Filesystem.Write failed to resolve on the dispatched
runner (undefined variable), while the srv1 observation module's identical call
had resolved by pool-membership coincidence — the Class B accidental-coverage
class exactly. All three modules using Filesystem.Write now import
extdeps.filesystem.filesystem_io explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…le, never Conflict, never satisfied

Operator emphasis: rightness over green. A ProbeHostKeyMismatch with Absent
expectation cannot honestly claim a contradiction — the wire's 'verification
failed' does not distinguish first contact from a changed key when nothing is
enrolled — so both verdict projections now split on expectation presence:
Present -> Conflict (a real contradiction to investigate), Absent ->
Inaccessible (transport never completed; enroll the fingerprint registry).
Both arms leave the cell NOT converged in the fleet fold; only the remedy
differs. Witnessed in both directions on both projections; all 8 probe
witnesses green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ate-receipt ambiguity wall; machine-id hazard checked

Endpoint: new gunbc.fleet_reach_endpoint (downstream of procurement AND the
probe, per bold-eagle's draft) projects SlotAssigned -> the cited CR1000A
reservation address via render_ipv4_address and SlotUnassigned -> no endpoint,
never a placeholder; the reach probe now transports to the resolved endpoint
(Sparks probe at 192.168.1.222/223; name-resolvable hosts pass through) while
receipts stay keyed by fleet identity — addressing is transport, not identity.

Machine-id hazard: --host self derives from kernel hostname, never machine-id
(note added stating both Sparks share one machine-id AND factory host keys, so
hostname is sound only after host-identity-converge); and the fold's
host_knob_standing no longer picks first-wins on duplicate receipts — 0 ->
Unobserved, 1 -> judged, >1 -> NotConverged naming the ambiguity, so a
two-machines-one-identity collision surfaces instead of one machine's receipt
satisfying the other. New witnesses: duplicate-converged-receipts read
ambiguous (RED-shaped) and Spark probe targets are modeled addresses.
9/9 probe + 6/6 fold witnesses green. No is-system-running usage anywhere in
the loop's health reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ismatch constructor split; receipt-dir re-homed to host_layout breaking the apply->emit->cli cycle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…remote/rev-parse ops, fleet_desired_observe standing, self-route revision gate + published revision cell

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
… srv2-4 via typed self-install plan (argv derived from the one timer authority, is-enabled readback), Sparks typed pre-enrollment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…er the typed argv seam, parsed by the backward read of the byte-locked grammar, and fed to the atomic fold; Absent/Unparseable stay typed evidence

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…r — Spark host keys are DISTINCT; machine-id is the shared non-anchor

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…is not TimerViaSelfInstall (srv1 landing would fork live_deploy ownership)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
briansrls added a commit that referenced this pull request Aug 9, 2026
… main push (implicit success() cascade) — status guard + RED control (#8085)

* Repair: push-to-main jobs silently skipped on every main push — condition now carries its own status guard, with a RED control

Measured latent (>=30 consecutive main pushes): heal_generated_artifacts skips on ordinary pushes; ci survives via its own !cancelled(); but a needs:[ci] job whose if carries NO status function gets GitHub's implicit success() ANDed in, and success() is false downstream of any transitively skipped job — so deploy_dashboard_srv1, advance_fleet_desired, and collect_fleet_convergence skipped while every clause in their condition was true (an empty-observation narrow: 'dependency skipped' rendered as 'condition false'). Receipt: run 31320306318 (ci=success, all three skipped); no deploy ran in the last 30 main pushes.

Fix at the one generating authority (ci_deploy_push_to_main_if): '!cancelled() && needs.ci.result == 'success' && <event clauses>'. ci.yml regenerated through the artifact gate.

RED control push_to_main_family_carries_explicit_status_guard: reds if the guard leaves the authority row, if any push-to-main job's condition drops it, or if the family goes vacuous. Verified discriminating by execution (reverting the row -> FAIL).

This is the focused repair arm under the CONVERGE-0 acceptance outcome (#8083 remains the acceptance carrier); items 1/2 of the receipt set become obtainable once this is on main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y

* Restore the authority row (RED-check revert had clobbered it before the first commit)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y

---------

Co-authored-by: Brian Searls <briansrls@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Brian Searls and others added 4 commits August 9, 2026 20:43
…e tests blanket sudo; the grant set is exact-argv — probe with sudo -n -l (run 31334610318 srv2 receipt)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…erged model

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…o-converge; read-only multi-principal probe for srv3/srv4 foothold measurement

- fleet_multi_principal_probe: read-only measurement (parent msg_e767a783) of whether the in-run fleet key authenticates as any pre-existing account (ubuntu/briansrls/ghrunner/root) on srv3/srv4 — hostname + sudo -n -l, no writes. Rides the existing WIF-authenticated probe job (one credential materialization).
- fleet_sudoers_bootstrap_scaffold: the parent's hand-applied srv2 dropin + srv1/srv2 gunbc-fleet accounts reclassified from 'legacy bootstrap' to DRIFT-TO-CONVERGE; dissolution = the membership spine reproduces srv1/srv2 standing Unchanged (a hunk would be the receipt that the hand-work was drift in substance). No further hand-applied host state; the real defect is live_deploy targeting srv1 only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…ncipal verification, failure-safe credential trap, key-identity check; + srv3/srv4 privileged foothold MEASURED

#1 UNOBSERVED PRINCIPAL NEVER WIDENS TO AMBIENT SSH: FleetProbeTarget = ProbeTargetReady{host,endpoint,remote_principal} | ProbeTargetRefused{cause: PrincipalUnobserved|EndpointUnobserved}. ProbeUserPending (Sparks) now REFUSES — no bare endpoint, no ssh attempt. srv5/srv6 rows are ProbeTargetUnobserved.
#2 REACH DOES NOT VERIFY PRINCIPAL: ProbeReachable carries observed_principal (second typed op 'id -un'); receipt carries requested_principal; identity Converged requires BOTH hostname AND principal to match (hostname-match-under-wrong-account is Drifted).
#3 FAILURE-SAFE CLEANUP: trap cleanup EXIT armed before the key touches disk (AGENT_STARTED flag), so any set -e abort wipes the key file and kills the agent; outer always() kept as defense in depth.
#4 KEY-IDENTITY VERIFICATION: before ssh-add, ssh-keygen -y | ssh-keygen -lf - compared to the modeled fingerprint authority; mismatch = ProbeCredentialIdentityMismatch, CONTACTS NO HOST.

MEASURED (run 31336668532, read-only multi-principal probe): the fleet key ALREADY authenticates as a PRIVILEGED account on both srv3 (ubuntu, sudo -n -l LISTED) and srv4 (briansrls, sudo -n -l LISTED) — so srv3/srv4 bootstrap is dispatchable through CI with no operator grant. ghrunner is nologin on both; root refused on both.

Collector consults the same target (refuses on unobserved principal). All probe/collector/host_phase/cli witnesses green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
Brian Searls and others added 2 commits August 9, 2026 23:10
…-job effects

The ladder correctly caught that probe_fleet_reach and collect_fleet_convergence carry the identical key-agent + cleanup scripts. That is not cacheable cross-job duplication — an ssh-agent lives in one job's process env and cannot be memoized across isolated jobs (same category as per-job deploy stages). Declared both as fresh effects (ci_fresh_effect_scripts); cleanup extracted to gunbc_ci_fleet_key_agent_cleanup_script (one authority, referenced by workflow step + gate). Both ladder witnesses green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
# Conflicts:
#	dag/gunbc/fleet_converge_cli.dag
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review August 10, 2026 00:21
… CI URL + requirement row derive from it

The secret pin lived in three places — SecretRef.version='latest', a hardcoded versions/1 in the CI curl URL, and pinned_version='versions/1' on the requirement row (§3 violation; and 'latest' contradicted the SECRET-VERSION-LIFECYCLE name-a-version rule). Now:
- fleet_automation_ssh_privkey_secret.version = '1' (the blessed version; single authority).
- fleet_automation_privkey_version_segment() and fleet_automation_privkey_sm_access_url() derive the path segment and the full SM access URL (project + secret + version) from that one SecretRef.
- ci_spec's key-agent curl + mismatch echo consume the URL/segment fns; fleet_host_key_enrollment's pinned_version consumes the segment fn.
Emitted ci.yml URL is byte-identical (proof the derivation reproduces the intended path). Rotating to v2 now edits one field.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
@gunbai-bot

gunbai-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Fixed in the latest commit (review 50872, RC — valid §3 finding). The secret version pin had three authorities; it now has one. fleet_automation_ssh_privkey_secret.version is set to the blessed 1 (no more latest, satisfying the SECRET-VERSION-LIFECYCLE name-a-version rule), and both downstream sites DERIVE from that single SecretRef: fleet_automation_privkey_sm_access_url() builds the full Secret Manager access URL from the ref's project + secret + version, and fleet_automation_privkey_version_segment() supplies the path segment. The CI key-agent curl + mismatch echo and the probe_identity_secret_access_requirement.pinned_version row all consume those helpers instead of hardcoding. The emitted ci.yml URL is byte-identical (.../secrets/fleet-automation-ssh-key/versions/1:access), proving the derivation reproduces the intended path; rotating to v2 is now a one-field edit. Witnesses green. — sent from wise-moth-430

@gunbai-bot
gunbai-bot Bot marked this pull request as draft August 10, 2026 01:07
…rator finding)

EVIDENCE INFLATION (operator scan): srv1/srv2 observed_principal='gunbc-fleet' was INFERRED from the ssh target on a run (31333011288) that predated the id-un op — a value naming an observation nobody made, inverting repair #2. Corrected by EXECUTION: re-ran the probe (run 31346260819, probed_at 2026-08-10T01:11:35Z) which runs id-un; observed_principal is now a real remote-process readback (=gunbc-fleet on both), so identity Converged is genuinely two-dimensional. Provenance + comment updated; static transcription now reflects a run that actually observed the principal (condition 4 -> receipt; condition 5 advanced).

§3 IMPORT FORKS (review 50880):
- ci_fleet_wif_auth_step: workload_identity_provider + service_account now derive from probe_identity_secret_access_requirement (was hardcoded literals beside the row it names as authority); rotation is one edit site.
- fleet_receipt_collector: fleet_reach_probe_targets imported from its definer gunbc.fleet_reach_endpoint (was the host_reach_identity_probe re-export — Class B).
- fleet_multi_principal_probe: clock_now_probed_at imported from gunbc.clock_read definer (was the host_reach_identity_probe re-export).

Emitted ci.yml WIF block byte-identical. All probe witnesses green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review August 10, 2026 01:20
@gunbai-bot

gunbai-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Addressed the operator's evidence-inflation finding and review 50880's three §3 forks at HEAD c97d65f:

Evidence inflation (the load-bearing one): srv1/srv2 observed_principal: 'gunbc-fleet' was inferred from the ssh target on a run that predated the id -un op — a value naming an observation nobody made, which inverts repair #2. Corrected by execution: re-ran the probe (run 31346260819, probed_at 2026-08-10T01:11:35Z) which runs id -un, so observed_principal is now a real remote-process readback (gunbc-fleet on both) and identity Converged is genuinely two-dimensional. Provenance + comment updated to a run that actually observed the principal.

review 50880 §3 forks:

  • ci_fleet_wif_auth_step now derives workload_identity_provider + service_account from probe_identity_secret_access_requirement (was hardcoded literals beside the row it named as authority). Emitted ci.yml WIF block is byte-identical.
  • fleet_receipt_collector imports fleet_reach_probe_targets from its definer gunbc.fleet_reach_endpoint (was the host_reach_identity_probe re-export — Class B).
  • fleet_multi_principal_probe imports clock_now_probed_at from gunbc.clock_read (was the re-export).

All probe witnesses green; PR back out of draft. Conditions 1–5 of the operator's seven now met (1 body-actuals, 2 desired-ref CAS at 0f9bf74, 3 collector-compiles via #8090, 4 id-un executed, 5 transcription reflects a real id-un run); 6 (fold to one verdict) and 7 (offline-return control) remain as the acceptance set completes. — sent from wise-moth-430

gunbai-bot Bot pushed a commit that referenced this pull request Aug 10, 2026
Operator correction taken: stop modeling, get to a live canary. This is the first
executable step and it settles the question no model can -- can the fleet
controller authenticate to srv5, and as WHICH principal.

Runs on a self-hosted fleet runner because the Sparks have no Actions runner, so
nothing bootstraps them from inside; authenticated remote application from a LAN
controller is the only mechanism. Uses the ONE permission measured as held
(versions.access on fleet-automation-ssh-key, run 31336704811).

READ-ONLY by construction: every command observes. Nothing is installed, no
authorized_keys is written, BatchMode=yes cannot prompt so a missing credential
fails instead of falling back to an interactive path, and the key is shredded in
an always() step. Token goes to a 0600 curl config, never argv.

The receipt states what a failure does NOT establish: it cannot distinguish
missing authorized_keys from absent account from sshd policy, and those have
different remedies. #8083's 'no verified managed principal' is a claim about our
records; this observes the host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…witness imports

Two findings on head c97d65f, both DESIGN-grounded and confirmed real:

(1) §3 fork — probe_identity_secret_access_requirement re-minted
secret_project/secret_name as literals beside the SecretRef authority in
fleet_ssh_access. Both now derive from fleet_automation_ssh_privkey_secret
(.project/.secret as String — the same casts fleet_automation_privkey_sm_access_url
already proves), so one place carries the secret identity. Rotating edits one row.

(2) Class B — fleet_host_key_enrollment_witness_test bound its cross-module
symbols by pool-membership coincidence (no import lines). Added the explicit
import block from the definers: LiveTreeDisposition/SubstrateInputsOnly from
v2.std.live_tree, HostIdentity from product.placement_supply, the enrollment
symbols from gunbc.fleet_host_key_enrollment, and the probe classifier + host-key
arms from gunbc.host_reach_identity_probe — matching the sibling
host_reach_identity_probe_witness_test import discipline.

Neither change touches emitted ci.yml.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
@cursor

cursor Bot commented Aug 10, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@gunbai-bot

gunbai-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Addressed cursor review 50886 (both findings, DESIGN-grounded and real) at 497719b:

  1. §3 secret-identity fork (fleet_host_key_enrollment.dag requirement row) — secret_project/secret_name were literals beside the SecretRef authority. They now derive from fleet_automation_ssh_privkey_secret (.project/.secret as String — the same casts fleet_automation_privkey_sm_access_url already proves), so the secret identity lives in one place and rotation edits one row. The pinned_version and CI fetch URL were already derived from that authority per review 50872; this closes the two fields that had drifted back out.

  2. Class B witness (fleet_host_key_enrollment_witness_test.dag) — bound its cross-module symbols by pool-membership coincidence. Added the explicit import block from the definers (v2.std.live_tree, product.placement_supply, gunbc.fleet_host_key_enrollment, gunbc.host_reach_identity_probe), matching the sibling host_reach_identity_probe_witness_test discipline.

Neither change touches emitted ci.yml.

— sent from wise-moth-430

Brian Searls and others added 3 commits August 10, 2026 02:47
… probe to enrolled-only known_hosts

Host-key audit (witty-raven-412 / bold-eagle-107 cross-check, 2026-08-10). Two
independent audits agree the blind-accept sites (StrictHostKeyChecking=no +
UserKnownHostsFile=/dev/null) are exactly the BMC/sshpass lane in
extdeps.exec.command, and the CONVERGE probe path (typed_argv_probe_over_ssh ->
ssh.Session.ExecArgv = `ssh {host} -- argv`) carries NONE of it. But the probe
did not pin UserKnownHostsFile to the enrolled authority either — it APPENDED the
fragment to $HOME/.ssh/known_hosts with no StrictHostKeyChecking, so host
verification rode the runner account's AMBIENT known_hosts of unaudited (OOBE /
prior-manual TOFU) provenance. Condition-4 fixed the principal half; the host half
was zero-dimensional.

Anchor search (fleet_physical_inventory, fleet_intent_network, enrollment
provenance) before concluding: srv1/srv2 have no cited reservation, no
procurement/serial, no external key authority — genuine TOFU, not the convenient
answer.

Two changes:
1. Mint HostKeyEnrollmentProvenance = ScannedFirstContact{scanned_from} |
   IndependentlyAnchored{authority}. authority is a REQUIRED NonEmptyStr, so
   "anchored but unnamed" has no constructor (construction over validation, §5 —
   no witness restates it). All 6 enrollment rows carry ScannedFirstContact
   (measured honest state); the srv5/srv6 DGX Sparks are the intended
   IndependentlyAnchored inhabitant (cited CR1000A reservation) but stay
   first-contact until their slot->box assignment resolves — asserting it now
   would encode the 50/50 guess DESIGN §5 forbids. Arm ready for that lane to
   flip without a widening edit (second-consumer discipline). Receipt carries the
   provenance label so "found" never reads as verification the row doesn't claim.
2. Pin the probe non-destructively: enroll_apply writes a SCOPED ssh config
   (`Host <fleet endpoints>` -> UserKnownHostsFile = enrolled-only fragment +
   StrictHostKeyChecking yes), replacing the ambient-known_hosts append. For fleet
   hosts ssh now consults ONLY this registry and refuses unknown/mismatched keys
   (a key change trips ProbeHostKeyMismatch). Scoped to fleet endpoints so
   non-fleet ssh is untouched even on durable HOME; marker-guarded so a repeat run
   appends once. Readback verifies against the pinned fragment.

Net climb: ambient-unaudited -> enrolled-authority-pinned, first-contact ceiling
carried by the provenance type. Banked run 31346260819 pre-dates the pin (rode
ambient); enrolled-anchored re-observe lands post-merge (probe jobs run on
main/dispatch, skip on PR), same window as the host-receipt item.

Green by execution: both fleet_host_key_enrollment witnesses return true; the
module compiles clean through analyses with the new provenance field. Pin is a
runtime .dag effect inside enroll_apply, not emitted ci.yml — no regen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…; the honest blocker is the unperformed anchor join

Factual correction (witty-raven-412, read against gunbc.dgx_spark_procurement).
My prior scanned_from prose said the srv5/srv6 slot->box assignment is "an open
operator decision" — false and would propagate. dgx_spark_router_binding_operator_allocation
(SPARK-ALLOC-LAND-0, 2026-08-07) records the operator DELEGATED the tie-break and
it was decided on ascending address order: srv5->192.168.1.222, srv6->192.168.1.223,
both carried as SlotAssigned rows. No 50/50 guess is outstanding.

ScannedFirstContact is still the correct label, for the right reason: no anchored
enrollment has been PERFORMED — nobody has enrolled a Spark host key, observed the
MAC at that address, and matched it against the cited CR1000A reservation.
IndependentlyAnchored would claim work that has not happened. The anchor is
AVAILABLE but UNUSED. This matters practically: bold-eagle is NOT blocked on an
operator decision (it was made three days ago), only on running the join — so the
Spark host-identity condition does not acquire a phantom blocker.

Prose-only; no type, value, or behavior change. Compiles clean through analyses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
…its SSH legs (same path as probe)

Real defect (cursor review 50900, §3/§5). collect_fleet_convergence_wet SSHs to
every fleet host but never called fleet_host_key_enroll_apply(), unlike
probe_fleet_reach_wet. The collector runs in its OWN isolated main-push job with a
fresh ephemeral HOME (CI overrides HOME per job) and WIF-materialized key but no
durable known_hosts — so the enrolled-only pin (StrictHostKeyChecking yes +
UserKnownHostsFile=fragment) that enroll_apply writes never landed there, leaving
the collector on the ambient-trust path the pin was written to close, and on a
different trust path than the probe.

Fix: call fleet_host_key_enroll_apply() FIRST in collect_fleet_convergence_wet and
carry its readback in the collect receipt, mirroring the probe entry exactly — one
enrollment authority, one trust path. Green by execution: the collector closure
compiles and runs to ExitSuccess with the enroll call (no effect/cycle error).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
@gunbai-bot

gunbai-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Addressed cursor review 50900 at 80aa480 — a real §3/§5 defect, confirmed.

collect_fleet_convergence_wet SSHed to fleet hosts but never called fleet_host_key_enroll_apply(), unlike probe_fleet_reach_wet. Its isolated main-push job has a fresh ephemeral HOME + WIF-materialized key but no durable known_hosts, so the enrolled-only pin (StrictHostKeyChecking yes + UserKnownHostsFile=fragment) never landed there — leaving the collector on the ambient-trust path the pin closes, and on a different trust path than the probe.

Fix: the collector now calls fleet_host_key_enroll_apply() FIRST and carries its readback in the collect receipt, mirroring the probe entry — one enrollment authority, one trust path. Green by execution (collector closure compiles + runs to ExitSuccess with the enroll call).

— sent from wise-moth-430

…operator request)

Operator: "can we simply have a single fleet convergence job — it's overwhelming
to see so many ... it can include all deployment/convergence i.e. srv1 deploy etc."

The ten former jobs (deploy_dashboard_srv1, collect_fleet_convergence,
advance_fleet_desired, probe_fleet_reach, scan_fleet_host_keys,
enroll_fleet_host_keys, observe_probe_identity, install_fleet_converge_timer,
enroll_runner_authorized_key, observe_srv1_tasks_preapply) collapse into ONE
`fleet` job, so a normal push/PR shows a single (mostly-skipped) row instead of
~nine. Each former job's work becomes a STEP carrying the event guard its old job's
if_condition held, applied via with_step_if:
  - main-push convergence steps (deploy/collect/advance): if github.event_name == 'push'
  - workflow_dispatch toolbox steps: if github.event_name == 'workflow_dispatch'
Job-level gate ci_fleet_job_if runs the whole job only on (main push + ci success)
OR workflow_dispatch — so PRs skip it as one row. Shared prelude + release-bins +
WIF/ssh-agent setup run ONCE (were duplicated across jobs — a materialization win),
key-agent cleanup runs always(). runs-on srv1; needs [ci]; permissions are the union
(contents:write for advance + id_token:write for WIF); deploy's blue/green
concurrency group carries to the whole job.

Tradeoff, stated: needs [ci] means manual workflow_dispatch fleet ops now wait for
the ci floor before running (the cost of one consolidated job; the convergence steps
must gate on ci success, which requires needs [ci]).

Cleanup: deleted the 11 now-dead per-op job builders, the unused
gunbc_ci_deploy_job_backstop_timeout_minutes and ci_fleet_credential_permissions,
and the orphaned ci_deploy_push_to_main_if / ci_workflow_dispatch_only_if
expressions (their skip-cascade rationale folded onto ci_fleet_job_if, which now
carries the !cancelled() + needs.ci.result guard the witness checks).

The skip-cascade wall is preserved: ci_fleet_job_if's push arm carries !cancelled()
+ needs.ci.result == 'success', so push_to_main_family_carries_explicit_status_guard
still holds (count is now 1 job, not >=2). Updated ci_deploy / ci_deploy_target_host
witnesses to assert deploy-as-step-in-fleet; consumer_jobs -> ["fleet"].

Green by execution: main_wet regen idempotent (ci.yml 10 jobs -> 5); the emitted
fleet job carries the correct per-event step guards; ci_workflow_witness_holds,
push_to_main_family_carries_explicit_status_guard, and both updated deploy
witnesses return true. Runtime main-push behavior validates post-merge (fleet
convergence steps run on main push only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168R31q15ZsPJyik18Roz3y
@briansrls
briansrls merged commit 32708d2 into main Aug 10, 2026
5 checks passed
@briansrls
briansrls deleted the session/wise-moth-430 branch August 10, 2026 05:27
briansrls pushed a commit that referenced this pull request Aug 11, 2026
…both DGX Sparks (#8073)

* Give the fleet an automation principal and a service principal

fleet-automation@gunbc is an SSH KEY COMMENT, not a POSIX username. The tree
had no automation account at all, so every modeled host access resolved
FleetAccountOperator -- the human -- and automation reached hosts as a person.

Adds FleetAccountAutomation (gunbc, uid/gid 1500) and
FleetAccountInferenceService (gunbc-inference, uid/gid 1501) to the single
POSIX-account authority. No Spark-local account enum beside it.

Privilege is one closed root-owned helper, not blanket sudo and not docker
group membership: the Docker control socket is root-equivalent, so joining
that group to fix a refused socket grants far more than deployment needs.
The serving principal's grant list is EMPTY by construction -- it runs a
model and answers requests, and never needs to change its host.

UID/GID pinned identically on both Sparks only because it was OBSERVED:
2026-08-09, both hosts carried exactly one uid>=1000 account, and both names
plus uids 1500/1501 were free on BOTH.

Recorded rather than smoothed over: these two rows are DESIRED and exist
nowhere, while every other PosixUser here is OBSERVED with a probe that
confirmed it. Their probes therefore record the availability check actually
run, not a fabricated existence check. That is gunbc#8068's authored-versus-
executed conflation one carrier over; the split is deliberately NOT invented
here, and dissolves when the apply creates the accounts.

25/25 existing principal witnesses PASS; module compiles clean.

* Revert the static-account commit; no host mutation may proceed against it

Rolls dag/gunbc/fleet_posix_accounts.dag back to main. The reverted commit
fused semantic identity with the operating-system label -- static usernames
gunbc / gunbc-inference and cross-host-equal uid/gid 1500/1501 -- which is
the same defect the hostname authority already corrected one carrier over,
reintroduced in a commit that argued for single authority.

It carried three further defects, all confirmed against the tree:

  - a duplicate declaration of the privileged-helper path, beside the
    existing gunbc_deploy_privileged_helper_path in the same module;
  - new accounts unreachable through fleet_posix_account_from_name;
  - availability probes placed in PosixAccountDeclaredProbe, whose
    invariant is exact equality with concat('id ', name) -- the rows would
    have FAILED it, and the 25/25 witness run reported as validation never
    enumerated the new arms, so nothing checked.

The working tree was rolled back earlier but the commit was never committed
or pushed, so the remote head still carried it while it was reported
reverted. That gap is the reason this is its own commit rather than a
silent amend.

* Split principal intent from account allocation from observation

The fourth route. Neither an optional fleet_posix_principal nor an
overlapping allocated-account enum: one semantic principal population, with
its fused result separated into intent -> occurrence -> allocation ->
observation -> grounding.

WHY NOT OPTIONAL. Absent cannot carry a remedy. Intent-absent,
allocated-but-unapplied, applied-but-unobserved, observation-contradicts-
allocation, drifted and grounded are six situations with six different next
actions, and a two-valued answer makes them one. Proven by execution: flipping
AccountUnallocated to report PrincipalUnobserved reds the unallocated control
while the unobserved control stays green, so the states are distinguished
rather than collapsed.

NameRequirements is a shared vocabulary -- audience, readability, permitted
semantic content, authority role, stability, uniformity, allocation method,
retirement -- and NOT a universal name type: DNS labels and POSIX login names
keep separate lexical authorities. Automation and service names are required
to carry NoDomainMeaning and to be IndependentPerOccurrence; the human label
is the stated exception, because a person must know and type it.

Consistency is four requirements, not one. The reverted commit asserted
UNIFORMITY (equal uid across hosts is one identity) when it needed
per-occurrence STABILITY. UID/GID are host-local numeric allocations in a
PosixIdentityDomain; equality across srv5 and srv6 is permitted, never
required, and never evidence of one principal. An inode records the number,
so a uid change is an ownership migration rather than a rename.

Both Spark automation occurrences are UNALLOCATED. No name chosen, no uid
reserved, nothing applied -- so nothing can render useradd, User=, a home
path, sudoers or authorized_keys, which is the point.

HumanAccessOnly is carried structurally on the human principal, because
host_identity_access and ci_deploy_access both resolve it today and that
fusion is live.

Consumes the existing gunbc_deploy_privileged_helper_path rather than
declaring a second one with the same literal.

7 new controls reach the new arms, 7/7 PASS, the wall mutation-tested.
Legacy witnesses unchanged: 25/25 grounded_principal, 11/11 ci_deploy.

* Delete the fabricated principal; the missing state was the defect

BLOCKING REPAIR. The recut removed a fabricated GroundedPosixPrincipal
(static gunbc, uid 1500) and replaced it with a DIFFERENT fabricated
GroundedPosixPrincipal: name UNALLOCATED-PRINCIPAL, uid -1, defended in a
comment calling it unreachable-by-contract while the type made it an
ordinary value every existing caller could consume into an effect, a path,
a comparison or a log without any match forcing acknowledgement. A
conspicuous poison value is still fabricated success.

WHY IT EXISTED, which is the actual finding: fleet_posix_principal takes no
host, and an account is per-host. There was no honest answer available, so
one was invented. The repair is the missing STATE --
PrincipalRequiresHostScope -- plus total per-account accessors for the four
legacy principals whose accounts really were observed on srv1/srv3. The
placeholder is deleted, not relocated.

Also repaired:
  - allocation standings are occurrence-bound; an unallocated answer for
    srv5 can no longer be consumed as srv6's with nothing to detect it;
  - allocation conflicts are preserved as their own outcome instead of
    being converted to unallocated, which erased both the attempt and the
    causes needing remediation;
  - conflict populations are first-plus-rest, so 'conflicted with no
    causes' has no constructor;
  - PosixAccountObservation carries its OWN occurrence rather than a copy
    of the desired allocation, and grounding compares the two subjects --
    a copied observation is now drift. Copying the allocation in would have
    made every observation agree with the plan by construction, which is
    the authored-versus-executed defect reproduced inside its own repair;
  - the inference service realization is Unselected rather than an
    unallocated persistent account, which had already chosen a /etc/passwd
    realization before the runtime exists to say whether a supervisor
    dynamic identity suffices.

STILL OWED BEFORE ANY APPLY, named in-carrier rather than silently missing:
process receipt, raw-output identity, decoder identity and timestamp on the
observation; separation of account grounding from access grounding, since
desired sudo grants currently ground an account whose sudoers may never have
been installed; POSIX lexical authority instead of NonEmptyStr; and the
semantic-content / readability axis split.

10/10 new controls (3 added: host-scope refusal, cross-host observation is
drift, service realization unselected). Legacy unchanged: 25/25, 11/11.

* Receipt-backed observation; account grounding split from access grounding

Completes the carrier repairs owed before any apply.

RECEIPT-BACKED OBSERVATION. PosixAccountObservation now carries a
ProcessReceipt, a raw-output identity and a timestamp. Without them any
caller could author a record asserting an account exists -- #8068's
authored-versus-executed defect at account grain, where the carrier proves
an account was DESCRIBED rather than that a command ran. The grounding fold
also stops synthesizing a probe with the literal 'observed' and carries the
observation's own timestamp.

ACCOUNT GROUNDING IS NOT ACCESS GROUNDING. The previous fold copied DESIRED
sudo grants into a grounded result, so an account whose sudoers was never
installed still read as subsumed -- it proved policy, not observed access.
ground_host_access separates four facts that can each hold while the next
does not: account matches allocation, key population observed, password
login refused, narrow privilege proven to execute. Only HostAccessGrounded
may authorize a serving deployment, which is exactly the srv5 canary
progression.

NAMING AXES SPLIT. An earlier revision made NoDomainMeaning and MnemonicOnly
peer variants, a false exclusion: a pronounceable arbitrary word is both a
human mnemonic and free of domain meaning, and that pair is precisely what an
automation label should be. NameSemanticContent and NameHumanFactors are now
independent, and a control asserts the combination the fused coproduct made
unrepresentable. NameIntent (what the name must be) is separated from
NameAllocationPolicy (how it is chosen and retired).

POSIX LEXICAL AUTHORITY. PosixLoginName lands in extdeps/access/posix.dag
beside the Open Group anchor, cited to the portable filename character set
and the leading-hyphen constraint -- deliberately NOT shared with DnsLabel,
whose grammar, length limits and authority differ. Scheme and scope
identities are branded; name_attempt is Nat, so a negative attempt has no
representation.

12/12 controls (2 added: account-grounded-does-not-imply-access-grounded,
unallocated-cannot-reach-access-grounding). Legacy 25/25 and 11/11.

* Author the Spark bootstrap secret producer (containers + IAM + material write)

The refs in gunbc.spark_bootstrap_credential were named with nothing creating
them. This is the producer, modeled on the live-proven
gunbc.assimilate.bmc_bootstrap_provision pattern against the same project.

Container provisioning and material write are separate patterns because their
idempotence differs: CreateSecret converges (409 on re-run), AddVersion does
not (every call mints a version). Collapsing them behind one verb would have
carried the BMC runbook's 'safe to re-run' promise onto a half where it is
false.

Payloads are pattern parameters, never data rows. Readback names the exact
version written rather than the moving 'latest' alias.

Token acquisition is a parameter rather than a direct
shell.GCloud.AuthPrintAccessToken() call, so this does not mint a third
authority beside gunbc.auth.access_token_source.ensure_access_token.

Compiles green over a 225-source closure; bindings evaluate to the real
least-privilege SA. NOT YET APPLIED: no admin access token is reachable from
this host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Correct a convergence claim refuted by execution; split the convergent op out

The authority note and the prior commit message both asserted that CreateSecret
converges -- 'a second run answers 409 and the desired state is reached either
way'. That was written from reading the API, not from running it. The reapply
was then executed and returned HTTP 409 ALREADY_EXISTS as a hard pattern
failure. The claim is corrected to what was measured.

The failure also exposed a design error: SetSecretIamPolicy (convergent, sets a
policy to a desired value) was bundled behind CreateSecret (not convergent).
Because creation runs first and aborts, a re-run never reached the bindings --
so the only operation that could repair drifted least-privilege bindings was
gated behind one guaranteed to fail on every host after the first apply. Split
into spark_secret_containers_create (one-shot, typed loud failure on re-run) and
spark_secret_iam_converge (re-runnable).

The remaining non-convergence carries a dissolve-on naming its next rung:
route CreateSecret through an UpsertClassification mapping 409 to Noop, at which
point the two patterns rejoin.

Measured 2026-08-09: first apply created both secrets and set both bindings;
reapply failed 409 on the first secret.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Record the executed re-runnability receipt for IAM convergence

spark_secret_iam_converge re-run against already-existing secrets succeeds
where the previously bundled pattern died on 409, which is the discriminating
receipt for the split.

Stated at measured strength: re-running leaves binding content byte-identical
but is NOT a no-op -- the policy etag advances each run, because the pattern
writes the desired policy rather than comparing and skipping. Re-runnable and
idempotent-in-result, not idempotent-in-effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add executing controls over the Spark grant shape

The producer module landed with applied receipts but no controls. A successful
apply proves the API accepted the operations and proves nothing about whether
the roles granted were the least ones sufficient -- an owner-level binding
applies exactly as cleanly -- so least privilege has to be asserted
structurally rather than inferred from a green run.

Five controls, 5/5 PASS by execution:
- least privilege: exactly two bindings, only the assimilator, allowed roles
- both intended roles present (control 1 alone passes if secretAccessor were
  granted twice and versionAdder never)
- escalated roles absent: the discriminating negative, red if someone resolves
  a permission error by reaching for admin
- the two secret identities are distinct: a collision would silently write both
  credentials into one secret as successive versions, since AddVersion never
  overwrites
- neither collides with the BMC secret sharing this project and SA, where the
  assimilator already holds versionAdder so the write would be permitted

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Recut Spark secrets as convergence subjects; delete the mutation patterns

Operator verdict on the interactive apply. The three patterns that dispatched
CreateSecret / SetSecretIamPolicy / AddVersion are DELETED rather than repaired,
because each was unconditional -- create with no prior Get, SetIamPolicy with no
GetIamPolicy and no etag (setIamPolicy REPLACES a policy, so unrelated bindings
on any secret carrying them would have been dropped), AddVersion with no attempt
identity -- and because leaving that shape callable is what made a manual apply
possible at all. Actuation belongs to an OIDC/WIF workflow that consumes this
model.

The population is FOUR host-scoped occurrences, not two global names. The
factory hotspot value differs per device, so one global hotspot secret would
have to hold two credentials, and since AddVersion never overwrites the second
write would silently become a prior version of the first. This is the same
host-scope error the POSIX principal split fixed one carrier over.

The two containers created interactively classify ObservedLegacyUnbound: real
starting-state observations, not accepted output, not the authority for any
occurrence, not edited or deleted from here. Neither carries a version.

The BMC assimilator principal is replaced by an unallocated
FleetCloudConvergencePrincipal. Binding a principal declared for the srv3 BMC
secret to Spark credentials made BMC assimilation the authority for Spark
provisioning -- a cross-domain expansion, and the nickname trap in its usual
form.

Six controls, 6/6 PASS. Note for the record: spark_bootstrap_credential's own
dissolve-on already required a receipt naming the exact version so an
interrupted retry reuses it rather than minting a second password. The deleted
AddVersion pattern violated a law written in the module it was extending.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Record the reported cloud census; split principal identity from population binding

witty-raven-412 reports (msg_1e6d1b42) the fleet key secret, a pre-existing
fleet-automation@ consumer, and a new fleet-cloud-convergence@ SA WIF-bound for
gunb-ai/gunbc. Recorded as REPORTED observations carrying their source, not as
measurements: this session has no gcloud, ADC, metadata server or credential
env, so it cannot verify any of them.

Two separations the report's phrasing would otherwise collapse:

Identity vs binding. The convergence SA exists and holds accessor on ONE key
secret, which does not reach the four Spark occurrences. Principal allocated and
population reachable are different facts; SparkPopulationBindingStanding keeps
them apart so 'the principal is set up' cannot read as 'the population is
bound'.

Identity join vs causal claim. The scratchpad key and SM version 1 share an
ED25519 fingerprint, so they are the same keypair. That does NOT establish the
local copy was materialized FROM that version: a fingerprint is symmetric, a
common origin or a copy predating the version yields identical evidence, and no
fetch was observed. Recording it as a materialization would assert an event with
no receipt.

Control 4 was also wrong in the other direction and is fixed: it asserted the
principal stays unallocated, which would have gone red the moment the correct
thing happened. The real invariant is that the Spark authority is dedicated
rather than borrowed, so it now reds on the BMC assimilator or the
fleet-automation consumer being substituted.

7/7 PASS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Container observe-before-create classifier and IAM membership diff

The two operations the deleted patterns did blind, modeled as decidable
classifications with no actuation.

Container: four verdicts, because absent, already-correct, present-but-not-ours,
and unreadable have four different remedies. Calling Create and reading a 409 was
an error arm standing in for an observation nobody made. A container that
ALREADY carries a version classifies Conflicted, never Converged -- AddVersion
never overwrites, so writing there leaves two live credentials for one
occurrence. An unallocated occurrence refuses rather than computing a probable
name, which is the reverse inference the carried-never-derived law forbids.

IAM: a delta against an observed policy, with an owned-universe parameter that
makes 'unrelated' decidable. A member outside it is preserved even when
undesired, because this model does not own it. A write with no observed etag
refuses -- setIamPolicy replaces rather than amends, so a write that cannot name
the policy it read silently overwrites a concurrent update.

Seven controls, 7/7 PASS, including the two discriminating negatives: an
existing version conflicts rather than converging, and an unrelated binding is
preserved rather than dropped (the exact defect the deleted pattern shipped).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Materialization attempt identity, receipt, and the hotspot source asymmetry

At most one AddVersion per attempt identity. Three attempt states, because never
tried, issued-but-unconfirmed, and confirmed have three different resolutions --
and the middle one is the point: it resolves by OBSERVING the store, never by
issuing a second AddVersion. A lost process outcome is not evidence the write did
not land, and treating it as such sprawls versions for one intended credential.

A confirmed receipt is discharged by an INDEPENDENT read agreeing with it, not by
having been written: a mismatched or unavailable readback refuses. A readback
naming the moving alias latest is Unavailable rather than Matched, since it is
compatible with another writer having landed in between.

THE ASYMMETRY, raised with the operator and modeled as a wall rather than a note:
the convergence shape says material is generated or fetched inside the admitted
run, which does not cover a FACTORY hotspot credential. That value is printed on
the device -- there is nothing to generate and nothing to fetch it from -- so its
only honest source is an operator observation through the sensitive-input
boundary. Generating one fabricates an observation of physical hardware that
every downstream check would accept while the credential was simply wrong, so the
pairing refuses. The administrator arm still admits generation, proving the
refusal is kind-specific rather than a blanket ban.

Five controls, 5/5 PASS. Nineteen on the module total.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Extract the repository-level GitHub-to-GCP federation as one authority

Operator directive: extract the federation and consume it from both domains
rather than copying the BMC rows into the Spark lane.

Nothing about a pool for gunb-ai/gunbc, an OIDC issuer URI, or the fleet secrets
project is a BMC fact. The tell was in the names: bmc_secrets_project_id held
gunbai-secrets, the FLEET secrets project, so a second domain had to either
import a bmc_ symbol for a non-BMC fact or copy the literal. Copying is how the
second authority gets minted, and the copy is invisible afterwards because both
sides read correct.

BMC now consumes the extracted rows; the values have one authority. Proven by
execution against BMC's own witnesses: 25/25 PASS across all three files
(bootstrap 4, token federation 7, delegation chain 14), including the ones that
would notice a changed value -- provider resource well-formed, principal set
pins the repo, least privilege, closed allowlist.

Residual naming fork left marked rather than silently carried: seven bmc_-prefixed
names still stand for repository-level concepts. Weaker than the value fork just
removed since no row can now disagree, but still a nickname; the dissolve-on names
the consumer rename that retires them.

Two process notes worth recording. The first run reported 0 PASS with no failures
-- that was ZERO ENROLLMENT from --roster-from-discovery with --entry, not
success, and a FAIL-grep would have read it as green. The second run failed 17/25
on 'no such function', because the new module was not in the entry closure: bare
cross-module references resolve only by pool-membership coincidence, so the fix
is explicit imports in both directions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Independence is a grant property: project into std.access, drop the global standing

Operator correction. Independence belongs to authorization policy, not to
usernames, uids, service-account counts, or how many credential values exist.

THE DISHONEST ARM, and it was mine: the carrier returned
FleetCloudPrincipalAllocated while its own note said the allocation was not
independently verified. Those cannot coexist -- the carrier reported a rung its
evidence did not establish, in the same commit that recorded the evidence as
second-hand. Split into Unallocated | Reported | Allocated; the live standing is
Reported until the WIF workflow reads it back, and a Reported principal cannot
authorize actuation.

SparkPopulationBindingStanding deleted. A global present/absent could not say
which operation on which resource, so 'the principal is bound' read as coverage
while naming nothing. Replaced by exact principal x operation x resource cells
over a scope carrying FrozenAtGrant vs TracksNamespaceMembership -- so adding
srv7 cannot silently inherit today's authority.

No second access kernel: std.access supplies AccessDecision, the typed refusal
causes and decision_meet. This module supplies only the domain operation
population, resource identity and scope rows. Action-not-granted and
object-not-granted stay distinct refusal coordinates rather than one blurred no.

Five new controls test independence as a GRANT property and mention no username
or uid: srv5 does not reach srv6; one global principal with one exact grant
reaches both admitted resources and not a third; AccessVersion does not imply
AddVersion and AddVersion does not imply DestroyVersion; a Reported principal
cannot actuate while an Allocated one can; an allocated principal is still bounded
by scope.

Full module roster 22/22 PASS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Model the login shell; record the ghrunner contradiction and the applied allocation

Three defects surfaced by witty-raven-412's live readback on srv1 and srv2.

SHELL WAS UNMODELED, and it is load-bearing: sshd refuses remote command
execution under a nologin shell, so an account can exist, carry the right key,
and still be unusable for the exact thing automation needs. PosixUser gains a
PosixLoginShell, modeled as an OBSERVATION rather than a required literal --
forcing a value would have invented three shells nobody read, which is exactly
how the uid rows beside them went wrong. An unobserved shell answers false to
posix_shell_permits_command_execution, because 'we did not look' is not 'it
works'.

THE FALSE RECEIPT, which is the sharper defect: fleet_posix_ci_runner_user
carries uid 1001/gid 1001 and a declared_probe stamping probe_command
'id ghrunner' on srv1. That exact command on that exact host returns 999/987, so
the row claims a probe whose own output contradicts the values beside it --
either it never ran or its result was discarded for authored numbers. A wrong
integer is visible; the provenance stamp is what made it look checked. Recorded
as a typed contradiction with source, NOT overwritten: replacing one authored
pair with another I cannot verify from here is a better-sourced guess, and
minting a PosixAccountObservation to hold someone else's reading would defeat the
ProcessReceipt wall that carrier exists to be.

ALLOCATION APPLIED: gunbc-fleet on srv1 and srv2, same label on every host,
because independence is a property of the grant and never of the string.
srv3/srv4 stay AllocationPending -- an unreachable host has no allocation rather
than a presumed one. Carried with the uid coincidence: gunbc-fleet took 1001 as
the next free id, which is what the ghrunner row wrongly claims, so two rows now
read 1001 for different accounts on one host. Harmless (uid is host-local and
never principal identity) but a reading hazard that raises the priority of
grounding ghrunner.

Also recorded: srv1's duplicate sudoers dropin makes effective privilege the
UNION of two files, so a grant readback reading one reports a strict subset and
would classify an over-granted host as exact -- a measurement hazard, not
untidiness.

46/46 across the three affected rosters (posix 12, managed_directory 9,
grounded_principal 25).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Remove two fabricated-success paths and join the receipt to itself

Review findings, all three mine and all three the class this module exists to
prevent.

AN EXACT VERSION ADDRESS IS NOT A PAYLOAD MATCH.
admit_spark_readback_version turned any non-latest string into
SparkReadbackMatched while consuming no AccessVersion response, no decoded
payload, no expected identity and no comparison -- fabricated success producing
the very verdict that discharges a receipt. Split into an address ADMISSION and a
separate compare_spark_readback taking both identities, so Matched is reachable
only when two supplied identities actually agree. The old control asserted the
defective behaviour, so the control and the bug agreed; it is rewritten to prove
admission alone yields nothing.

THE AT-MOST-ONCE WALL WAS ADVISORY. spark_materialize_once let any caller mint
SparkMaterializeOnce without consulting the attempt record -- the wall in the
prose, the bypass in the interface. Deleted. A first write is now reachable only
through SparkAttemptJournaled, a durable record written BEFORE the effect issues,
so a crash between journal and issue lands in IssuedUnconfirmed and resolves by
observation rather than a second AddVersion.

THE RECEIPT DID NOT JOIN ITS OWN FIELDS. A receipt naming exact_version 7 whose
readback verified version 8 produced Noop, because nothing compared them. Two
fields that must agree and are never checked are two facts, not one.

24/24 PASS.

NOT addressed in this commit and still open from the review: the parallel grant
system (no AccessRequest/AccessPolicy construction, no authorization_profile, no
effect_grant reach, a second Secret Manager operation enum disagreeing with the
extdeps surface), missing workflow/revision/purpose/duration/use/approval context,
IAM diffing at flat-member rather than role x member x condition grain, the
consumer-derived secret demand, the local spark_srv5/srv6 identity roster, the
legacy container carrier holding no receipt, and every POSIX fail-open. No srv5
apply is safe from this head.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Read-only WIF effective-access observation (temporary push trigger)

The first thing on this branch that changes observed reality rather than its
description. Authenticates via GitHub OIDC to WIF as fleet-cloud-convergence@,
mutates nothing, and emits the effective-permission map.

What it decides: fleet_cloud_principal_standing currently returns
CloudPrincipalReported -- an allocation another session reported and this one has
never verified -- and the model refuses to let a Reported principal actuate. This
run either grounds it with a receipt or refutes it.

Expected standing is EffectiveAccessIncomplete: the principal is reported to hold
accessor on ONE secret, so the Spark containers should refuse. An unexpected
permit is the more interesting outcome and classifies Overbroad. Both are real
receipts; neither is a failure.

Read-only by contract. The testIamPermissions calls are POST because that is the
API's shape; they return the granted subset of the supplied permissions and change
no state. No AccessVersion call is made, so no payload is read, and the token is
never echoed.

Declared scaffold on two axes, each with its dissolution named in the file: the
YAML and shell are hand-authored where the project's rules say a foreign-executor
workflow should be emitted from .dag, and the push trigger is temporary so the
observation runs without a manual press.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Ground the cloud principal by execution; record measured effective access

FIRST OBSERVED RECEIPT ON THIS BRANCH. GitHub Actions run 31335059188 at head
6b79d061c4a: OIDC to WIF issued a token for fleet-cloud-convergence@ with no key,
no pasted credential and no gcloud, then probed its own permissions read-only.

CloudPrincipalReported -> CloudPrincipalAllocated. The allocation another session
reported is now confirmed by an observation this session can cite, so a principal
my own model refused to let actuate has a receipt behind it.

EFFECTIVE ACCESS, MEASURED: secretmanager.versions.access on
fleet-automation-ssh-key, and NOTHING on either Spark container.
testIamPermissions returned an empty object for both, which is a positive
statement of no permission rather than a 403's ambiguity between denied and
absent -- the reason the probe asks from the resource side instead of inferring
from status codes. versions.destroy and secrets.setIamPolicy were probed
explicitly and denied everywhere, so the grant is narrow BY MEASUREMENT and there
is no Overbroad finding. Standing is EffectiveAccessIncomplete with the seven
missing cells enumerated rather than guessed.

Two facts nobody had recorded: this principal cannot read secrets.get anywhere,
so it cannot independently verify the legacy containers (what I know about them
came from an operator token, not this path); and the Cloud Resource Manager API
is disabled on the project.

CONTROL REPAIRED, second instance of the same shape: the reported-cannot-actuate
control asserted against the LIVE standing, so grounding the principal by
observation would have RED it -- a control forbidding progress rather than
forbidding the defect. It now takes a Reported fixture, since the invariant is
about that arm.

25/25 PASS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Correct the testIamPermissions overclaim; take the token out of argv

Two defects in code I landed hours ago, both mine.

THE INSTRUMENT CANNOT SUPPORT AN ABSENCE CLAIM. The prior receipt asserted that
an empty testIamPermissions response is 'a positive statement of no permission
rather than a 403 ambiguity', and concluded there was no Overbroad finding and an
exact missing-cell population. That is backwards: the API returns the empty set
for a resource that does NOT EXIST, is documented as a permission-aware-UI helper
rather than an authorization check, and may fail open. So an empty response
carries exactly the denied-or-absent ambiguity a 403 carries, and I had claimed
it RESOLVED that ambiguity.

Standing is now EffectiveAccessPartiallyObserved: one positively observed cell,
seven requested-but-not-returned (never 'missing'), and overbreadth explicitly
UNOBSERVED -- no policy was enumerated, and a probe cannot find a binding it did
not ask about. WIF issuance is split into its own standing, because the
credential path working is not evidence about what the credential may reach.

THE TOKEN WAS IN ARGV. The workflow passed the bearer header as a curl -H
argument, placing the credential in curl's command line where any process on the
runner can read it from /proc -- the same argv exposure this session already
recorded for a CLI flag, repeated in the file whose own comment claimed the token
was never exposed. It is now written once to a 0600 curl config by heredoc,
passed with --config, and shredded in an always() step.

Also recorded in the workflow rather than left implicit: it emits log text, not a
typed artifact receipt, so the .dag rows are hand-transcribed from it -- a second
authority that closes when the workflow is emitted from the model.

26/26 PASS, each witness named individually in the untruncated log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Repair the GCP IAM surface; reconcile at cell grain; split the two principals

Commit 1 of the directed sequence. Four separable defects, one measurement.

EXTDEPS DID NOT CARRY THE OPERATIONS THE MODEL NEEDS. Added GetSecret, a
secret-scoped GetIamPolicy, and TestSecretIamPermissions. SetSecretIamPolicy was
wrong three ways at once: it took a bare binding list with NO etag and returned
only an etag, so it silently dropped every binding the caller did not author, had
no optimistic concurrency to refuse a concurrent edit with 412, and never showed
the caller the policy it actually installed. It now carries the full GcpPolicy
including the etag it was read at, carries updateMask, and returns the RESULTING
policy so a write is checked against what landed rather than what was sent.

THE PROBE CANNOT PRODUCE ABSENCE, SO THE TYPE DOES NOT LET ANYONE WRITE IT.
SmPermissionProbeCell has Held and Undetermined and NO third constructor. This is
the same overclaim corrected in the previous commit, moved one rung up the ladder:
there it was a prose correction to a receipt, here the invalid state has no
representation, so an "exact missing-permission set" is unwritable rather than
merely discouraged. GetSecretIamPolicy is the only instrument that settles it.

NO ROLE-PERMISSION TABLE IS AUTHORED. Predefined role contents are Google-owned
and change without us, so role_grants_permission reads a RolePermissionAuthority
with an explicit Unobserved arm and returns a three-valued verdict rather than
false. The citation behind the observed arm is a new gcp.IamRoles.GetRole -- its
own service, because roles live on iam.googleapis.com while token minting lives on
iamcredentials.googleapis.com, and hanging it off the credentials service would
have silently requested the wrong host.

THE IAM DIFF WAS AT THE WRONG GRAIN. It compared flat member strings, which cannot
express the question a policy answers. One member under secretAccessor and under
admin is ONE entry at member grain, so demoting a principal reads as "already
present, nothing to do" and the admin binding survives. Reconcile now works on
GcpIamBindingCell -- role x member x condition -- with condition equality part of
cell identity, and ownership asked per cell so owning (M, accessor) does not imply
authority to remove (M, admin).

TWO PRINCIPALS, FORCED BY MEASUREMENT. Run 31336704811 at da9d440f8f4 probed
fleet-cloud-convergence and got back exactly secretmanager.versions.access;
secrets.getIamPolicy did not come back and secrets.get returned a live 403. That
principal provably cannot read a policy, so overbreadth stays unobservable until a
grant administrator exists. The data plane now reaches payloads and never policy;
the control plane reaches policy and never payloads. The refusal subject is DERIVED
from the grant -- the prior literal "FleetCloudConvergence" would have located an
administrator's refusal against the wrong subject the moment the split landed.

The grant is a thin profile over std.authorization_profile's EffectRequest and
finally carries the context it needs to be adjudicated at all: workflow, revision,
purpose, duration, max uses and approval. FrozenAtGrant names the roster identity
it was frozen against, so "frozen" is checkable instead of a label.

34/34 on the spark suite, 12/12 on the secret-manager suite, rosters matched.
Nothing actuates. Owed and NOT claimed here: a discrimination check on the
cell-grain controls, consumer-derived secret demand, workflow generation from .dag,
and the POSIX fail-opens.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Give the two BMC callers the read-modify-write my extdeps change required

CI fix. The break was mine and the class was one I had already been warned
about: I reshaped SetSecretIamPolicy in extdeps and swept only the module I was
looking at, leaving two real callers reading a field that no longer exists --
bmc_bootstrap_provision and bmc_wif_canary_bootstrap, both on bind.ok_etag.

WHAT THE BREAK EXPOSED IS WORTH MORE THAN THE BREAK. Both callers were passing a
bare binding list with no etag: a blind whole-policy overwrite, which is exactly
the defect the extdeps repair exists to make unwritable. Reverting to green would
have restored it. So the compile error was the wall working as intended, and both
sites now observe the policy first, merge, and write with the observed etag --
the compare-and-swap the API actually requires.

merge_policy_bindings lives in extdeps.cloud.gcp.iam beside GcpPolicy, not in a
consumer: every caller needs the same preserve rule, and three copies of it would
drift. Its bound is stated where it is defined rather than assumed -- it replaces
at ROLE grain, so a caller authoring a role takes over that role's member list.
Cell-grain preservation needs the caller's owned-cell universe, which that
function does not have and must not guess.

Both files also gained explicit imports for GcpPolicy/merge_policy_bindings.
They previously carried no imports at all and reached everything as bare
references, which resolves only when some unrelated module in the closure happens
to drag the definer into the pool.

VERIFIED BY CONTROLLED COMPARISON, not by the diff looking right. Same binary,
same command, red head b7c0d19b41a vs this tree:

  ok_etag errors   2 -> 0
  Empty errors    32 -> 32   (pre-existing, in complexity_accumulator_copy, not mine)
  diagnostic lines 79 -> 77  (exactly the two removed, nothing new)

Both runs exit 1 on those pre-existing Empty errors, so this is a delta claim and
NOT a whole-tree-green claim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Defork IAM reconciliation onto one cell-grain mechanism in the GCP authority

I introduced this fork two commits ago and it is removed here rather than left to
propagate into the demand work.

THE FORK. The CI repair added merge_policy_bindings, which replaced the whole
member population of every role the caller authored. That was a SECOND
reconciliation semantics beside the role x member x condition mechanism the Spark
module already carried -- the dual representation this lane exists to remove --
and its failure mode was concrete rather than theoretical: an unrelated member
under a role the caller also uses is dropped with no diagnostic, because whole-role
replacement cannot see members it does not author. Its own note admitted it could
not preserve those members. A helper that has to disclaim the property callers
need is not a shared mechanism; it is a second one wearing the first one's job.

THE REPLACEMENT lives with GcpPolicy, not in a consumer, because every caller
needs the same read-modify-write and three copies would drift. reconcile_gcp_policy
takes an explicit GcpIamOwnedCellUniverse and: flattens observed bindings to cells,
adds missing desired owned cells, removes stale cells ONLY inside the exact owned
universe, preserves every unowned cell including another member under an authored
role, regroups cells into wire bindings so an unauthored cell survives the round
trip, and carries the observed etag so the write stays a compare-and-swap.

GcpPolicyAlreadyConverged is its OWN arm rather than a delta that happens to be
empty, so a caller cannot write an unnecessary setIamPolicy and read the resulting
etag bump as progress.

Both BMC callers and the Spark module now consume that one mechanism; Spark's
local cell predicates delegate to it instead of duplicating it. merge_policy_bindings
is DELETED, not parked beside the replacement -- a climb deletes the machinery it
obsoletes rather than accumulating next to it.

Also fixed, both caught by the compiler rather than by assumption: an annotation I
placed inside a declaration body (annotations are modeled at module-item grain
only), and a twelve-line comment block still describing merge_policy_bindings after
I deleted it -- prose outliving what it names.

VERIFIED. Spark suite 34/34, roster matched. Whole-tree dag compile against the
pushed head, same binary and command:

  hard diagnostics  67 -> 67   (mid-state with my annotation bug was 69)
  diagnostic lines  77 -> 77
  Empty errors      32 -> 32   (pre-existing, not mine)

No new diagnostics. Both runs still exit 1 on the pre-existing Empty errors, so
this is a no-regression claim and NOT a whole-tree-green claim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Derive access demand from selected effects; extdeps owns operation identity

The desired access population is no longer authored. It is a projection of the
effects this workflow will actually perform.

WHAT WAS WRONG. SparkCloudGrant.operations x SparkCloudScope.resources was a
hand-maintained roster crossed into cells: a second inventory of what the code
needs, kept beside the code, so it drifts -- and the cross product granted every
listed operation on every listed resource, so a plan that reads secret A and
writes secret B acquired write on A and read on B for free.

EXTDEPS OWNS THE UPSTREAM FACTS, following the github_effect_required_permissions
precedent already in tree rather than inventing a shape. SecretManagerEffect
carries the EXACT resource it acts on; secret_manager_effect_operation mints the
OperationRef beside the operation declaration it names, so no consumer authors
service/operation strings; secret_manager_effect_required_permissions states each
operation's permission requirement, because which permissions an operation needs
is Google's contract, not a workflow's guess.

DEMAND IS ONE CELL PER (selected invocation, required permission), with the
resource taken from the effect. There is no roster left to cross. Every cell
carries source_effect, so "why is this permission requested" is answered by a plan
step a reader can find.

TWO FAIL-CLOSED ARMS, both deliberate. An operation whose permission projection is
empty yields DemandIncomplete and BLOCKS actuation rather than contributing no
demand -- an empty requirement would read as "needs nothing" and admit the effect
with no grant at all. And one incomplete step refuses the WHOLE derivation, because
a partial population read as total is how an ungranted effect slips through.

THE CEILING STAYS NORMATIVE AND SEPARATE, now keyed on the real OperationRef.
Selected code does not authorize itself: control 4 asserts BOTH halves -- a
selected DestroyVersion produces a visible demand AND is refused. The
enum-keyed ceiling is demoted to a shim delegating to the ref-keyed authority, so
deleting SecretManagerOperation is mechanical rather than a risky half-migration.

A SIDE FINDING, kept because it was load-bearing: this witness had NO imports and
reached its entire subject by bare reference, resolving only by pool coincidence.
Adding one explicit import scoped resolution and 39 of 40 controls died at once
with "no such function" -- the transition is all-or-nothing, there is no partial
state. The subject module is now named explicitly, which also means those 34
pre-existing controls no longer pass by coincidence of pool membership.

Also hoisted ten annotations out of match arms and function bodies (annotations
are modeled at module-item grain only) -- my debt from the two-principal split.

VERIFIED. Suite 40/40, roster matched. Whole-tree dag compile vs pushed head:
hard diagnostics 67 -> 57 (the ten hoisted annotations), Empty 32 -> 32, nothing
new. Both runs still exit 1 on the pre-existing Empty errors, so this is a
no-regression-plus-ten claim, NOT whole-tree green.

NOT DONE, not claimed: SecretManagerOperation deletion, consumer-derived secret
population, role realization from observed permissions, approval derivation, and
controls 6-12. Control 6 is deliberately unwritten -- every current operation has a
permission mapping, so testing it would require a planted fake, and a control that
only passes against a fabricated input is not evidence about the real projection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Derive secret demand from named consumers, not from the credential roster

THE FOUR-CREDENTIAL POPULATION WAS NEVER A DEMAND POPULATION.
spark_credential_population is a roster of credentials that COULD exist for a
host; it says nothing about whether any workflow still needs one. Reading it as
demand made a post-OOBE factory hotspot credential an automatic blocker on
serving, because "four occurrences, none allocated" reads as four outstanding
obligations whether or not anything consumes them.

SecretDemandStanding derives from a CONSUMER -- a named workflow plus the
operation it will perform. Four arms, because these have different remedies and
collapsing them is how a break-glass credential becomes live authority:
Active (named consumers), RetainedForRecovery, RetiredAfterOobe, Unobserved.
Only Active yields routine demand.

TWO JUDGEMENTS, stated as judgements rather than derivations. RetainedForRecovery
is INACTIVE: a break-glass credential should be reachable by an explicit recovery
workflow, not by the standing grant every serving run carries. And the serving API
credential is Unobserved rather than Active: the pinned artifact and its
acquisition path are not selected yet, so an Active row would fabricate a consumer
that exists in no plan -- it is the credential most likely to become active next,
and it is still not asserted early.

An Active arm with an empty consumer list claims consumers exist while naming
none, so secret_demand_is_active refuses it rather than letting the arm's NAME
stand in for the fact.

SCOPE, stated so this is not overclaimed: this ADDS the demand authority beside
the existing population rather than deleting it. Nothing yet consumes
SecretDemandStanding in an acceptance condition, so the model now permits the
correct answer rather than enforcing it, and the old four-occurrence roster is
still present for the controls that reference it. On the guarantee ladder this is
mitigatable moving toward mechanically preventable -- not structural, and NOT
"the serving blocker is removed".

VERIFIED. Suite 42/42, roster matched. Whole-tree dag compile: 57 hard
diagnostics and 32 Empty, both exactly baseline -- no regression. Still exit 1 on
the pre-existing Empty errors, so this is no-regression, not whole-tree green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Name the BMC no-op bypass instead of letting it read as protection

Found in review, and it is mine from the defork commit. Both arms of the
reconcile match in the BMC wrappers return the policy, so a CONVERGED policy is
still written: GcpPolicyAlreadyConverged is computed and discarded at the only
call site that mattered. An arm that exists and is consumed nowhere is worse than
an absent one -- it reads as no-op protection while every run writes, and an
unnecessary setIamPolicy advances the etag, so a convergent re-run looks like it
changed something.

Not repaired in place: the caller is a `node bind:` pattern invocation, which
cannot branch on a verdict to skip its own effect. Making the write conditional
requires the call site to become an effect-plan step that can be selected or not,
which is the dissolution trigger recorded in both files.

Named rather than silently left, because the alternative is a wrapper that looks
correct. Control 11 (zero-delta reapply is a Noop) therefore does NOT hold at the
BMC call sites today, and I am not claiming it does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* srv5 canary step 1: observe managed-principal reach, read-only

Operator correction taken: stop modeling, get to a live canary. This is the first
executable step and it settles the question no model can -- can the fleet
controller authenticate to srv5, and as WHICH principal.

Runs on a self-hosted fleet runner because the Sparks have no Actions runner, so
nothing bootstraps them from inside; authenticated remote application from a LAN
controller is the only mechanism. Uses the ONE permission measured as held
(versions.access on fleet-automation-ssh-key, run 31336704811).

READ-ONLY by construction: every command observes. Nothing is installed, no
authorized_keys is written, BatchMode=yes cannot prompt so a missing credential
fails instead of falling back to an interactive path, and the key is shredded in
an always() step. Token goes to a 0600 curl config, never argv.

The receipt states what a failure does NOT establish: it cannot distinguish
missing authorized_keys from absent account from sshd policy, and those have
different remedies. #8083's 'no verified managed principal' is a claim about our
records; this observes the host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add branch-scoped push trigger so the canary probe can actually run

workflow_dispatch resolves against the default branch, where this workflow does
not exist, so the dispatch 404'd. Same constraint and same remedy as
spark-cloud-observe. Scoped to this branch, not a required check, gates no PR,
with deletion on canary completion as its dissolution trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop the canary's own harness from killing its primary observation

srv5:22 OPEN and AccessVersion http=200 with a 419-byte key: reach and credential
materialization through WIF on a fleet runner are now proven by execution. The
principal question went unanswered because GitHub Actions runs `run:` under
bash -e, so the first ssh returning non-zero aborted the step before any line
printed.

That is the probe's own harness converting its primary observation into a crash --
a failed login is exactly the data this step exists to collect, since the question
is WHICH principal srv5 admits, not whether one succeeds. Each attempt now
captures its exit code and continues, and the step's success no longer depends on
any login working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The probe fabricated its own exit codes; make the principal verdict readable

THIRD harness defect in this probe, and the worst kind: it manufactured the exact
field a reader uses to judge success. I wrote OUT=$(...) || true followed by
RC=$?, so $? captured the status of `true` -- every row reported rc=0, including
ghrunner, ubuntu and nvidia which were plainly refused with Permission denied.

What the run text does support: those three are genuinely REFUSED (publickey,
password). gunbc-fleet is UNRESOLVED -- no denial, but no id output either, and a
successful login would have printed a principal and uid. Absence of a denial is
not a login, and I am not recording it as one.

Two repairs. Exit status is captured with if/else so it is the ssh status.
And the remote command prints an explicit PRINCIPAL= marker, because bare `id -un`
output was indistinguishable from an empty capture -- 'no denial and no output'
could not be told apart from a successful login whose text was lost. A login now
counts as ADMITTED only when that marker is present; anything else is
REFUSED-publickey or UNRESOLVED, which are different states with different
remedies.

Standing so far, by execution: srv5:22 reachable from the fleet LAN, and the fleet
key materializes through WIF on a runner (http=200, 419 bytes). The admitted
principal remains UNOBSERVED.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Re-probe with briansrls: I omitted the principal my own evidence names

Correct catch against me. My candidate list -- gunbc-fleet, ghrunner, ubuntu,
nvidia -- came from naming CONVENTION, not from evidence. briansrls is the
principal my own reports point at twice: the Sparks' factory OOBE account, and the
principal under which wise-moth measured the fleet key ACCEPTED on srv4. Choosing
candidates by convention while holding evidence that names a specific account is
how a 'nothing admits the key' verdict gets manufactured.

Now probing briansrls, root, dgxuser plus the original four, across BOTH srv5 and
srv6. Same read-only construction, same corrected harness -- nothing installed,
BatchMode cannot prompt, explicit PRINCIPAL= marker, real exit status.

RECONCILIATION OWED regardless of outcome: an earlier report of mine claimed
fleet-key login verified end-to-end on both Sparks, key-only. That and 'no
principal admits the key' cannot both be true. If briansrls admits it, the earlier
claim was probably right and my candidate list was the defect. If everything still
refuses, the earlier claim was wrong and I need to establish which observation was
mistaken -- that matters more than the unblock, because one of them is a false
verification sitting in the record.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Canary step 2: observe srv5 serving-relevant facts before pinning anything

Step 1 answered its question: briansrls is ADMITTED on srv5 AND srv6 with the fleet key
(PRINCIPAL=briansrls UID=1000, read back from the remote process), every other candidate
REFUSED-publickey. My earlier "no managed principal admits the key" was a defect in the
candidate list, not an observation of the hosts.

This step observes what a pinned serving process would be pinned TO: accelerator, driver,
container runtime, interpreter, privilege, free space, memory, existing listeners, and any
serving process already running. Read-only; installs nothing.

Every probe prints an explicit ABSENT marker on absence, because an empty capture is
indistinguishable from a failed one -- the exact conflation that produced the false negative
in step 1.

* Fix empty-vs-absent in the two fields a serving canary relies on; interrogate :11000

`pgrep -af ... | head` left the || arm unreachable because head succeeds regardless, so a
no-match and a failed probe printed the same empty string -- in existing_serving and hf_cache,
the exact pair used to claim srv5 was not already serving. pgrep's own exit code separates them:
1 is "ran, matched nothing", >1 is "did not run". hf_cache now tests for the directory rather
than inferring absence from empty du output.

:11000 has an unidentified listener on the host about to run an inference server. Privilege will
not name the process, so this asks it what it is over HTTP -- status, body, /v1/models, and the
Server header. Silence records as a typed unknown; open-but-not-HTTP, closed, and probe-failed
are three different states and none of them is "unrelated".

* Probe serving-toolchain feasibility on GB10 aarch64 before the install step

:11000 answered: an NVIDIA-branded web UI (HTML, nv-dark), not an OpenAI-compatible endpoint --
/v1/models redirects to /. It is occupied and localhost-bound, so the serving port must not be
11000. The two repaired fields now carry real observations: existing_serving=NONE-MATCHED and
hf_cache=ABSENT-no-such-dir, both distinguishable from a failed probe.

GB10 is aarch64 Blackwell, where "a serving runtime has an installable build" is a genuine
question rather than a formality. Assuming a wheel exists is how an install discovers its blocker
halfway through a mutation, so this observes CUDA toolkit, ollama, uv, pip, venv, torch and the
resolvable vllm candidate list read-only first. A candidate list is not an install; it only
separates "no build exists for this platform" from "a build exists and installing is next".

* Canary step 2/3: install pinned vLLM user-unit on srv5, read back the LOADED identity

First mutation in this lane. Scope is derived, not preferred: sudo -n NO and the refused docker
socket are one cause with two closures (no system unit, no container), so user scope is what
remains. venv + vllm==0.26.0, systemd --user unit with lingering, bound to 127.0.0.1:8000 --
8000 measured free, since :11000 is NVIDIA's own localhost UI. Writes no system unit, no
authorized_keys, nothing outside $HOME.

THE READBACK IS NOT /v1/models. vLLM echoes the model id it was launched with and the revision
is not in that response at all, so comparing it to the pin compares a string against itself --
it would answer identically if a different revision, a stale cache, or an unrelated checkpoint
had loaded. That is a receipt supplying its own denominator.

The independent fact is produced by the LOADING: the HF loader resolves a revision to
hub/models--*/snapshots/<SHA>/ and the serving process mmaps the weight files from it, so the
SHA in /proc/<pid>/maps is the path the process actually opened rather than an argument I typed.
Open fds and the cache listing are recorded alongside it. /v1/models is still printed, labelled
DEPENDENT, for contrast only. If no independent fact is observable the step says
READBACK=UNESTABLISHED rather than falling back to the tautology.

Three separate facts kept separate, because a resolve is not an install and an install is not a
load: pip resolution (already observed), pip install rc, and a standalone import check, so an
installed-but-broken build is a named outcome rather than a mysterious startup failure. A GB10
initialization failure is reported as a measured outcome; no substitute serving stack gets
selected here.

* Derive serving install scope from observed privilege; import HostIdentity from its definer

The user-unit scope now exists in the model as a DERIVATION rather than an unexplained choice.
srv5 was measured before anything was installed: sudo -n refuses and the container socket refuses
the admitted principal. Those are not two independent limits but ONE cause with two consequences,
since a system unit and a container both need an escalation this principal lacks.

Modeled as derive_install_scope(HostPrivilegeObservation) -> ServiceInstallScope because an
unexplained scope invites a later reader to "upgrade" it to a system unit, silently reintroducing
the out-of-band human step the workflow path exists to remove. Stated as a consequence, that
upgrade is visibly a change to the OBSERVATION: it requires granting privilege first.
UserUnitScope carries its ground so the reason travels with the decision, and
InstallScopeUnobserved is a separate arm so an unmeasured host cannot default into user scope and
read as derived.

Also fixes a pre-existing resolution defect this exposed: HostIdentity was used in five places
here and imported nowhere, resolving only by pool-membership coincidence -- the Class B bare
cross-module reference DESIGN.md records. A narrower single-entry compile fails on it. Now
imported from its definer, product.placement_supply, and the module compiles with zero errors of
its own.

* Register the pip install as a declared scaffold; discriminate derive_install_scope both directions

The serving dependency install is now a registered Scaffold with a named dissolution trigger
binding to gunbc.host_effect_realize host_effect_apply, rather than debt described in a message.
It dissolves when the venv construction, pinned package install, and systemd --user unit write
are each a typed effect through host_effect_apply and the YAML carries no shell leaf. The Spark
is a runtime-present site already reachable over an authenticated transport, so it has no
foreign-executor or pre-runtime-bootstrap excuse for rendering bash. Debt that is only described
is debt nobody schedules.

derive_install_scope was asserted and compiled but discriminated by nothing, which is the inert
mechanism class: wrong in exactly the way it exists to prevent, silently. Three controls, 3/3
PASS by execution against a roster of 3:
  - observed srv5 facts produce the user-scope arm
  - no privilege must REFUSE system-unit and container installability
  - privilege actually SELECTS the arm (sudo -> system, socket admits -> container)
The third is load-bearing: the first two both pass on a function hardwired to return user scope,
because srv5's observed facts ARE the no-privilege case. Without it the derivation is
indistinguishable from a declaration and the function name does the body's work.

The loopback bind is recorded as deliberate and explicitly NOT part of this scaffold, so a later
reader dissolving the shell does not widen the endpoint with it.

* Close endpoint auth, lock the resolved closure, verify loaded revision on EVERY start

Three defects the audit surfaced, fixed in the order they constrain srv6.

FLOATING TRANSITIVE DEPS. A pinned model and a pinned vllm still leave torch and everything under
it resolving to whatever pip picks that day, so two hosts can differ while every pin reported is
identical -- latent on one host, a real divergence the moment a second exists. The install now
captures the resolved closure from the host that actually works (pip freeze -> constraints.txt)
and emits it in the log for srv6 to install against. Replaying onto srv6 first and reconciling
after is the one ordering that makes a difference indistinguishable between the day and the host.

ENDPOINT AUTHENTICATION. Loopback bind plus the fleet key on the transport is real protection and
is why the bind does not move, but "authenticated model" is an ENDPOINT property and the transport
cannot stand in for it -- two different threats. vLLM now runs with --api-key. The key is
generated on the host into a 0600 EnvironmentFile and never leaves it: not in the repo, not in CI
logs, not in argv, not in the unit. Secret Manager is not its home because this principal holds
versions.access only and cannot AddVersion.

PER-START VERIFICATION. An initial-load readback describes a process that may since have
restarted, and Restart=on-failure can return the server on whatever is in cache with nothing
distinguishing that from the pinned revision. verify-loaded-revision.sh runs as ExecStartPost on
every start, reads the same independent fact (the snapshot path the process actually mmapped, not
/v1/models, which would echo the launch argument and pass regardless), and REFUSES -- failing the
unit -- when the loaded revision disagrees, is unobservable, or no process exists.

* Move verification out of the start path; stop the restart loop; verify cache after 20 SIGKILLs

The hardening I added to close the restart gap is what broke the load. verify-loaded-revision.sh
ran as ExecStartPost, and with Type=simple systemd bounds ExecStart plus ExecStartPost by
TimeoutStartSec (90s default) -- so a verifier waiting for a multi-minute weight load was
terminated and the LOADING ENGINE WAS SIGKILLED WITH IT, then restarted into the same race twenty
times. A guard that blocks the thing it guards: the mirror of an absorbing fallback, and only
visible by execution. I reasoned about what the check would PROVE and not about what it would DO
to the lifecycle it was attached to.

Verification now runs as its own oneshot unit ordered After= the service, which is the right fix
for a principled reason rather than a practical one: inside the start deadline, "load failed" and
"verifier timed out" produce the same journal outcome, so the verifier's failure is
indistinguishable from its subject's -- a state-space conflation at the lifecycle layer. Ordered
after, those two states separate permanently. It still refuses on mismatch, via ExecStopPost
stopping the service on nonzero exit. TimeoutStartSec is deliberately NOT raised: removing the
ExecStartPost removes the deadline problem, and inflating it would blunt a signal a genuinely hung
start should still trip.

Prior generations are stopped and reset-failed before the unit is rewritten, so the new definition
does not race an old process for the port and the GPU.

CACHE INTEGRITY GATE. Twenty starts each SIGKILLed partway through a 14.19 GiB checkpoint read i…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant