Apply the host-wide executor grants from the converge spine, and read the kvm grant back

[gunbai-bot]

What was wrong

/dev/kvm membership for the runner executor was authorized and never applied. gunbc.runner_host_grants runner_host_wide_refused_operations emits EnsureSupplementaryGroupMembership{job_user, kvm}, runner_host_sudoers_content renders PERMISSION to run its argv, and gunbc.runner_microvm_host_ready explicitly disclaimed applying it — granted by one module, disclaimed by another, executed by neither. srv2 reads kvm=writable only because an operator typed the rendered argv by hand on 2026-09-05; srv1 reads present-not-writable, so its micro-VM boot probe cannot reach an image verdict. That is the rostered class gunbc.recurring_failure_mode capability_arrives_outside_the_converged_path.

Where the applier went, and why not where the brief said

My brief said to wire the applier into the microvm_host_converge mode. The rostered class forbids exactly that, by module name: "A lane wired the existing applier from gunbc.runner_microvm_host_ready — the disclaiming module — as its own CI run step, which is exactly the arm this ruling forbids" … "NOT an applier bolted beside the disclaiming module." Its reason is substantive: an apply reached outside the spine inherits none of fleet_converge_plan_held_lease, so two applies against one baseline could both proceed. I escalated; the operator ruled for the spine and corrected the brief.

So gunbc.fleet.fleet_converge_plan host_wide_grant_apply_lines emits runner_host_wide_refused_operations into the full-host apply script, beside the fabric-cell effects, rendered through executor_privileged_operation_converge_command — the same value the sudoers line is rendered from, so what is permitted and what is executed cannot drift into two spellings. The grants therefore run under the generation flock with every other host mutation. Correction: an earlier version of this description said an operator reads each argv in plan.txt. That is wrong — the grant argv is emitted into apply.sh, and the witness asserts against artifact.apply_shell; nothing here establishes it as a line in plan.txt. PLAN produces a content-addressed bundle of both files, so the argv is reviewable before apply, but the reviewer must read apply.sh to see it.

They resolve through runner_host_spec_for, not runner_host_deploy_for: these rows depend on no runner population — which is why runner_host_wide_refused_operations takes the spec — so resolving through the deploy would deny srv2, a rostered host with a refused width, the membership its own installed sudoers already authorizes. A host with no roster row contributes a stated refusal comment and no grant.

The readback, and why it is not test -w

A supplementary group enters a process's credentials at exec. The converge that applies the grant, and every process already running, still read present-not-writable — so a test -w /dev/kvm readback would refuse a host that had just converged correctly, forever. The readback asks the account database instead (id -nG <login>, which names the login explicitly rather than reporting the caller's own stale credential set), and the standing carries it as its own typed fact:

• KvmGroupEnrolled / KvmGroupNotEnrolled / KvmGroupEnrolmentUnreadable — the third arm is separate because "we could not ask" and "the grant did not land" send an operator to opposite remedies.
• Readiness splits in two: firecracker_host_ready admits enrolment (the converge's question — effective at the next incarnation), firecracker_host_ready_in_this_process requires writability (the boot probe's question, because it opens the device itself). Gating the probe on converge readiness would make it attempt a boot it cannot perform and report a VMM error as an IMAGE verdict.
• The receipt states the grant is effective for executor incarnations started AFTER this line, so present-not-writable beside enrolled is not read as a reason to restart a live slot. Nothing is restarted, masked or stopped; a JIT slot picks the group up on its own when its current job ends.

Evidence

claim_batch on both witness modules, all PASS:

• runner_microvm_host_ready_witness (7): readiness_needs_a_reachable_kvm_device_and_the_modeled_version, enrolment_carries_a_host_whose_converging_process_cannot_yet_open_the_device (the discriminating one — it reds if readiness requires this process to open the device), an_unreadable_membership_neither_carries_nor_reads_as_a_missing_grant, the_in_process_predicate_refuses_exactly_where_the_converge_predicate_carries, the_group_readback_matches_a_whole_name_and_not_a_prefix (a substring test would accept kvm-admin), plus the two pre-existing claims updated for the new field.
• fleet_converge_plan_witness (2): the_full_host_apply_script_carries_the_hosts_own_executor_grants asserts the EXACT elevated argv '/usr/bin/sudo' '-n' '/usr/sbin/usermod' '-aG' 'kvm' 'ghrunner' for srv1 and that a non-rostered host contributes no usermod line at all.

The fold's output was also read directly: host_wide_grant_apply_lines(host: srv1) yields the converge state dir, both runner tree roots, the kvm grant and the daemon reload, each fully elevated and quoted.

What this does NOT do

It does not attribute a capability to the run that placed it. The class's next-rung trigger — a standing that REFUSES a capability it cannot attribute — is untouched; this makes the modeled path the one that delivers, not the one that can tell whether it did. The roster row records both halves.

Scope limit of the route control, stated because it was demonstrated

both_microvm_steps_pass_the_selected_host_to_the_binding asserts over the authority — the Step values in fleet_converge_workflow.dag. It establishes that the authority delivers the expected host to both modes. It does not establish that the emitted workflow does, and those are two different propositions.

This is not hypothetical: the claim was green at 1af9f21b9d0 while the emitted YAML was wrong. Composing main conflicted in .github/workflows/fleet-converge.yml, I took the base side to settle the conflict and did not re-run the regen, so the authority carried the binding and the artifact GitHub actually runs did not — and a control that never reads the artifact cannot see authority-to-artifact drift by construction.

Asserting over step values rather than emitted text is still the right call for what that claim is for. The proposition it cannot carry is carried by the generated-artifact gate, which caught this drift on the real head and failed the step. Without this paragraph a reader sees 18/18 plus a route-repair claim and concludes the emitted workflow is covered — the exact inference that was false one commit ago.

The generalisable form, since it cost a red here: taking the base side of a generated projection is correct for settling a merge and insufficient as the repair. The authority and its projection then disagree, and only the projection is executed. The regen is the second half of the resolution, not optional cleanup.

Not yet run on srv1

Under the spine ruling the actuation is a full-host converge plan+apply on srv1, which is broader than the single mode the brief named. Asking before actuating.

🤖 Generated with Claude Code

[gunbai-bot]

Fixed in ba2479a. review 68771 is right: the relocation note named fleet_converge_host_wide_grant_lines, which resolves nowhere — the emitter is gunbc.fleet.fleet_converge_plan host_wide_grant_apply_lines. It was the module's one pointer at the relocated applier, so a reader following it found nothing; DESIGN §3 is exactly on point and the name is now the one the tree carries.

— sent from wise-wren-164

[gunbai-bot]

All three side-chat findings are fixed in 8142469. Each was real; taking them in order.

(1) kvm_reachable short-circuited on KvmDeviceWritable. That answers a question about the NEXT incarnation from the CURRENT one — a process that inherited the group before the grant was revoked still holds an open path to the device, so the host read ready while the login was no longer enrolled at all. Converge readiness is now answered entirely from durable host state: device exists AND delegated AND enrolled AND installed. A writable device is evidence for firecracker_host_ready_in_this_process and nothing else. New claim a_writable_device_does_not_carry_a_login_the_database_no_longer_enrols covers both shapes you named — writable+not-enrolled and writable+enrolment-unreadable — false for converge, true in-process.

(2) Enrolment plus presence did not prove the next incarnation can open the device. Took the delegation observation rather than the rename, because the rename would leave the standing unable to tell the two explanations apart. KvmDeviceDelegation = Delegated | Contradicted{expected_group, observed_group, group_writable} | Unreadable, observed by joining the device's group name with its group-write bit, and converge readiness requires it. Contradicted carries what was seen because the remedy differs by which half failed. The mode comes from a new PathMode operation on extdeps.tools.coreutils_stat (%a, its own probe — joining it onto the %u:%g:%U:%G line would let one unreadable half destroy the other); what the digits MEAN stays with extdeps.access.posix, which gains file_mode_of_octal_text as the declared inverse of file_mode_octal rather than a parser authored beside the consumer. Your point about Ubuntu's udev default being upstream desired state rather than a readback is recorded on the type.

(3) The header. Confirmed your reading of the fold before touching it: slot_apply_lines emits mask --now only for units the retirement admission authorized as surplus and removes a tree only after an independently reobserved RetirementComplete; the inner headers name both honestly. The OUTER header said "ADD only". It now names all three dispositions and states that no desired runner is stopped, restarted or reinstalled. The new claim asserts the retired wording cannot come back.

Evidence: 9 claims PASS in runner_microvm_host_ready_witness and 3 in managed_directory_witness under claim_batch. fleet_converge_plan_witness could not be resolved locally — refused twice with MemoryStallRefusedPageThrash on a host at 99/125 GiB, which is cannot-tell, not red — so its two claims including the new header claim are executed by the required witnesses lane rather than by me.

— sent from wise-wren-164

[gunbai-bot]

The two fleet_converge_plan_witness claims have now actually executed — closing the evidence gap I left open, and correcting something review 68883 accepted from me in good faith.

The correction. I wrote that these two claims were deferred to "the required witnesses lane", and review 68883's closing note accepted that as "cannot-tell reported as cannot-tell — the lane is the wall". There is no such wall. On this head .github/workflows/witnesses.yml has exactly three jobs: compiler (a cargo build of the gunbc bin), clippy, and witnesses, which is an aggregator whose only step reads needs.compiler.result and needs.clippy.result. It invokes no claim runner, and grepping all of .github/workflows for claim_executor/claim_batch matches only fleet-converge.yml, the host-converge workflow on workflow_dispatch. So a green witnesses check on any PR in this repo currently means the compiler builds and clippy is clean, and says nothing about any claim. That is escalated separately as a repo-wide gate question; here it just means the deferral was not a valid disposition and I owed the execution myself.

The run. Worktree HEAD 7982284574e, clean tree:

claim_batch --source-root dag --source-root src/v2 \
  --entry dag/test/claim/fleet/fleet_converge_plan_witness_test.dag \
  --functions the_full_host_apply_script_carries_the_hosts_own_executor_grants,the_slot_family_header_names_every_disposition_the_fold_can_emit

PASS the_full_host_apply_script_carries_the_hosts_own_executor_grants
PASS the_slot_family_header_names_every_disposition_the_fold_can_emit
rc=0
[witness] the_full_host_apply_script_carries_the_hosts_own_executor_grants: cpu=56ms eval_steps=21059 subject=3091ad96caebe6e9
[witness] the_slot_family_header_names_every_disposition_the_fold_can_emit:  cpu=37ms eval_steps=19287 subject=35709e0a4182af59
[resolve-summary] 1 resolve(s) in 697673ms wall; 2 witness(es) in 95ms wall
per-shard-peak-rss — 11.8 GiB

Five earlier attempts were OOM-killed at ~10–12 GiB peak while the session host was contended; this one ran once the slice cleared. All 14 claims on this PR have now executed — none are deferred.

— sent from wise-wren-164

[gunbai-bot]

Correcting my earlier comment on this PR, which said "no PR check executes witness claims". That was true of the head it was written against and is not true as an absolute — but the accurate statement is worse, not better, so I am stating it rather than quietly dropping the point.

(This replaces a mangled version of this comment: I passed the body through a double-quoted shell string and its backticks were executed, which ate several of the values and spliced in a 404 blob. The figures below are the real ones.)

A floor lane now exists on the merge path and it does build the witness executor and run witnesses. On this PR's run (35505981911, job 106065764780) its steps are "Build the compiler and the witness executor" → "Nominal witnesses (one prepared subject, one fold)" → "Generated artifacts match their authorities", all success, 16m49s.

It also fails open. On run 35505868144, job 106065468152 (head 217f5afff, not mine), I fetched the job log myself — 1150 lines — and read:

required-ci: floor refused: subject=1e469fe1625337cc modules_resolved=1859 modules_excluded=0
required-ci: lane=witnesses phases_run=2 phases_failed=1
required-ci: FAILED PHASE floor refused: ...

…while the API reports the job conclusion=success and step 8 conclusion=success, and the count of [witness] lines in the entire log is zero. 1859 modules resolved, no witness executed, a phase explicitly printing FAILED PHASE, exit 0.

So the two readings are not in conflict: the lane executes witnesses when nothing refuses, and does not fail the step when something does. The gate is real on a clean head and absent exactly when something is wrong — which is the worse of the two shapes, and is §5's "a failure arm must refuse, never widen" inverted onto the gate itself. A green floor is therefore not evidence that any claim ran.

One method note, since it nearly cost me the finding: my first [witness] count of 0 came from a 1-line file. gh api .../logs refuses to emit terminal escape sequences without --allow-escape-sequences, so the grep ran over the refusal message and returned a confident zero that happened to match the expected answer.

This does not change the evidence position for this PR: all 14 claims here were executed locally and reported with their run output, not deferred to any lane.

— sent from wise-wren-164

[gunbai-bot]

Re-ran every claim on this PR with a binary of known provenance, because the earlier greens had none.

A sibling lane (keen-bear-791) retracted a 45/45 after finding it had run claim_batch from a shared /cargo-target build that was 17 seed commits stale — stale enough that it could not see the checker that refuses a defect on main, so it reported passes a current binary refuses before any claim evaluates. Same tree, different compiler.

Applying that to my own receipts, they failed:

• claim_batch: path /cargo-target/release/claim_batch, mtime 2026-09-20 04:19 when I checked it, 05:29 afterwards — it was relinked under me between my check and my last run. Build head: unrecoverable (I did not build it; it has no --version).
• gunbc (the one that produced my "regen makes no changes" result): --version reports a35c7cede3ee77fb6c03c12f5269d3b310f0f328-dirty — a real ancestor of my head but 14 commits behind, with one seed commit in the range (730a5c56ca3, v2 manual tests: stop referencing test fns #11579), and -dirty with uncommitted changes from a worktree I cannot inspect. "No changes" is exactly the answer a compiler that cannot see your rows would give.

So I withdrew both and rebuilt from my own head into a private CARGO_TARGET_DIR (the shared one is what relinked under me).

Provenance — identical for both binaries:

path	…/scratchpad/tgt/release/{gunbc,claim_batch}
build time	2026-09-20 11:26:55 UTC
build head	611088d7ed154980c432afe3fee63e0f045a379d

gunbc --version prints that sha back with no -dirty, and it equals the worktree HEAD, clean, at run time.

Results, both unchanged:

• Regen (docs_projection_gate regen): rc=0, no changes to the tree. The projection fixed point was not an artifact of the stale binary — it reproduces. I had prepared for this to come out the other way.
• Claims: one invocation, 3 entry groups, 14 witnesses, rc=0 — 14 PASS, 0 FAIL. [resolve-summary] 3 resolve(s) in 256983ms wall; 14 witness(es) in 80ms wall, per-shard-peak-rss — 14.1 GiB.

The verdicts did not move. What changed is that they are now worth something: a witness verdict is only as current as the binary that produced it, and on a shared target directory reaching a stale one is the default rather than the exception.

— sent from wise-wren-164

[gunbai-bot]

Both findings are real and both are fixed in 8e4b95f41d4. review 69146 and review 69224 identified the same root and they were right: the binding I landed had no executing route.

microvm_host_binding_wet reads FLEET_CONVERGE_EXPECTED_HOST, and both steps that invoke the entries carried env: none — so microvm_host_converge and microvm_boot_probe would have exited ExpectedHostAbsent on every dispatch. The readback this PR exists to deliver could not start. I verified it rather than taking it on report: fleet_converge_workflow.dag:1049 and :1192 both had env: none, the emitted YAML had no env: on either step, and there is no job- or workflow-level env in that file. That is §3c's consumption question answered in the negative, and §5's specification-without-execution — the witnesses supplying binding values is correct per §3, but the real-path half was dead, not merely unexercised.

Fix. Both steps now set FLEET_CONVERGE_EXPECTED_HOST: ${{ github.event.inputs.host }}, the way every sibling entry that reads the variable already does, and the workflow was regenerated from its authority (generated_artifact_gate main_wet_one, rc=0) rather than hand-edited.

Second surface, same root. The step prose and the operator-facing mode description both still said this mode "refuses the kvm grant by name" — true of the disclaiming version, and now false in the direction that matters: a reader would not know the grant has an executor at all. Both now say the grant is applied by the full-host apply spine and read back here, refusing by name when it has not landed.

Control. both_microvm_steps_pass_the_selected_host_to_the_binding asserts over the step values, not the emitted YAML text — the text is a projection and the defect was in what the projection was given. Both modes are checked, because they fail independently and a control over one would have left the other dark.

One thing I did not fix, reported rather than silently worked around. The natural home for that control is test.claim.workflow_dispatch_input_witness, which already imports both step functions. That module does not resolve at this head: a pre-existing workflow_trigger_entry(trigger: fleet_converge_workflow.on[0]) at line 153 now produces Optional<Coproduct(WorkflowTrigger)> where Coproduct(WorkflowTrigger) is declared. It is untouched by this PR (my diff there was +29 appended lines, which I reverted). A control placed in a module that cannot resolve is a dark control — exactly the defect being repaired — so the claim went into fleet_converge_plan_witness instead. The broken module is a separate finding and is not fixed here.

Evidence at this head with the exact-head binary (built 2026-09-20 14:51:29 from 21ea5879105, no src/ delta since): 4 PASS in fleet_converge_plan_witness including the new control, rc=0.

— sent from wise-wren-164

[gunbai-bot]

Retracting one claim from my previous comment. I said test.claim.workflow_dispatch_input_witness "does not resolve at this head" and reported it as a separate finding. That was a statement about my stale tree, not about the module.

origin/main line 162 already carries the repair — a guarded match fleet_converge_workflow.on.first() { Present { value: trigger } => … Absent => false }, with an annotation noting that list indexing is total and typed T?, so passing on[0] where a required WorkflowTrigger is declared was the witness asserting a cardinality the language never granted. My branch predated that commit, so my tree still had on[0] at line 151 and refused. I read an error out of my own checkout and reported it as a property of the repo; that's the wrong direction of inference and I should have checked main before writing it down.

And the class was already rostered — I did not discover it. gunbc.recurring_failure_mode.witness_that_fails_to_compile_is_absent_rather_than_red is on main, and its SPECIMEN A is this exact wall: it names 978f6aaa342 (the Optional-at-Required refusal), names this module and the on[0] site typed T? by check_index_access_node, and explains the invisibility as DeclinedOutsideGateClosure — the module sits outside the required gate closure, so the refusal never reaches the floor and surfaces only from a local claim_batch run.

What stands from that comment: the claim lives in fleet_converge_plan_witness, and that placement is right for the reason the row itself gives — a control in a module that cannot resolve is a dark control. The binding-route repair and the prose repair are unaffected.

Current main is now composed (1af9f21b9d0). That merge moved the seed — 5 .rs files, ~1495 insertions in src/v1/stage0 — so the exact-head binary behind my earlier 4 PASS is stale at this head by the provenance rule, and I am not offering that verdict here. Rebuilding and re-running the targeted set; I'll post the head with full provenance.

— sent from wise-wren-164