witnesses.yml: ask for the rustfmt component, and stop the run step declaring it consumes nothing

[gunbai-bot]

The symptom

gunbc#8860 went red on a merged head while the witness floor passed 10130/10130, failed=0, zero FAIL identities. A different phase in the same step failed:

required-ci: phases_run=3 failed=1
required-ci: FAILED PHASE regen refused: normalize candidate wt_b.rs:
             spawn rustfmt: No such file or directory (os error 2)

The regen phase writes the emitted mirror as a fixed point of the formatter — the repair DESIGN records for the two-consumers-normalizing-one-artifact contradiction — so it must spawn rustfmt. witness_floor_toolchain_step never asked for it: its with block carried only cache: false. The job inherited whatever the runner's resolved default toolchain happened to carry, so it passed on some runners and failed on others with no relationship to the diff under test.

gunbc.fleet_workflow_steps ci_setup_rust_step already passes this same key to this same action. This step was the one that did not.

The obvious reading is wrong, and this repository already paid to learn it

I first diagnosed this as "that runner has no rustfmt" — which is what the error says and is not what it means. gunbc.host_toolchain_components toolchain_completeness_authority_note records the identical error on the identical host class on 2026-07-25: rustfmt was present on the machine; the job principal carried two rustup toolchains and defaulted to the one without it.

A component is a property of a toolchain, not of a machine.

So "install rustfmt on that host" is the fix that converges green while the gate keeps failing. The fix belongs in what the workflow asks for.

Why nothing caught it

witness_floor_run_step was declared capability_neutral while its regen phase spawns rustfmt. A step that consumes a capability while declaring it consumes none is invisible to the closure check by construction — the check was honest and had nothing to catch, because the consumer said it consumed nothing. RustfmtCapability already existed in the vocabulary and was simply unused here.

Four changes, not one

The components line alone fixes today's symptom and leaves the false declaration standing for the next tool that gets spawned.

1. witness_floor_toolchain_step passes components: rustfmt
2. its role becomes provides_only([CargoCapability, RustfmtCapability])
3. witness_floor_run_step declares consumes_only([RustfmtCapability])
4. witnesses.yml regenerated through the artifact gate

CargoCapability is deliberately not declared on the run step: it invokes the prebuilt claim_executor binary and does not shell out to cargo. Listing it would trade one false declaration for another.

Evidence

The live consumer workflow_capability_closure_witness the_live_witness_floor_job_closes_its_capabilities folds the job this module actually emits, not a transcribed fixture:

tree	verdict
this branch	true
provider drops RustfmtCapability, consumer still declares it	false

The RED is the point. This PR exists because a check was structurally blind, so a green alone would prove nothing — it would repeat the same error one level up. The control confirms the closure is genuinely discriminating for this capability.

Also: 0 blocking errors, and regen leaves the tree clean.

Why a second workflow moved

fleet-desired.yml also gains the component. That is correct rather than collateral: gunbc.fleet_desired_admission_workflow reuses witness_floor_toolchain_step, so one authority feeds both emissions. The change is additive — adding a provided capability cannot falsify any consumer's declaration — so no closure elsewhere can break on it.

This PR is the fix; #8877 is the instrument. Neither substitutes for the other.

Stated here so a reviewer meeting both does not read either as covering the other's ground.

This PR asks for the rustfmt component and makes the run step declare that it consumes it. That is the repair: it is what makes the component present, and it is what lets the capability closure refuse the shape that produced the failure.

#8877 adds a step that records the toolchain environment — PATH, the cargo bin directory, rustup which rustfmt, the toolchain list. That step fixes nothing and refuses nothing. It exists because spawn rustfmt: No such file or directory (os error 2) reports the syscall and nothing about the environment that produced it, so the failure destroys its own evidence and every occurrence is exactly as unreadable as the last. If this fix turns out to be incomplete, or the failure returns from a different cause, the probe is what makes the next occurrence diagnosable on its first sighting rather than its third.

So: the probe landing is not evidence that the rustfmt question is handled — this PR is. And this PR landing does not retire the probe, because a fix removes one cause and an instrument reads whatever the next one turns out to be.

[gunbai-bot]

What this PR fixes, and what it does not — written before attempt2 lands

It fixes a declaration that was lying. The toolchain step really did not ask for components: rustfmt, and the required-ci run step really was bound capability_neutral while the process it launches spawns rustfmt. Both are now correct: the toolchain provides RustfmtCapability, the run step declares it consumes it, and workflow_capability_closure_witness reds if the provider drops it (verified — returned false on that mutation).

It is NOT established to fix the spawn failure. Run 32556977241 on this head still refused with spawn rustfmt: No such file or directory (os error 2) — and the toolchain step in that same run logged info: component rustfmt is up to date, so the component was present in the active rustup toolchain and the bare Command::new("rustfmt") PATH lookup in required_regen_host.rs normalize_generated_source_attempt still found nothing.

Two readings I held tonight are both dead. It is not "one host lacks rustfmt": srv2-01 ran main green at 05:20 and this head red at 06:26, and its slots are serialized so there is no overlap. It is not the runner lottery either, for the same reason.

A rerun cannot settle it, and I am saying so before its result exists. gh run rerun does not pin the runner; attempt2 was reassigned from srv2-01 to srv1-01. That varies the exact quantity in dispute. A red attempt2 would be informative — the failure would then be on two hosts. A green attempt2 would tell us nothing, and I will not read it as vindication. A rerun proves recovered, never environmental; relocation is not a control.

So the state of this PR is: correct modeling that does not close the failure it was prompted by. That is still correct modeling — a red rerun would not make the declaration true again, and a green one would not make it the fix. The spawn failure is being pursued separately with an instrument, because ENOENT reports the syscall and nothing about the environment that produced it, and the next red will be exactly as unreadable as this one until that changes.

— sent from fierce-lynx-647

[gunbai-bot]

First probe output from #8877, and it bears on this PR

The toolchain probe in #8877 ran green for the first time (run 32585532283). It produced the baseline that did not previously exist, and one discrepancy worth eliminating here.

On srv2, runner srv2-03-1787416981-758643:

PATH=/home/ghrunner/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin
which -a rustfmt      -> /home/ghrunner/.cargo/bin/rustfmt        (rustup shim, present)
rustup which rustfmt  -> /home/ghrunner/.rustup/toolchains/1.93.0-aarch64-unknown-linux-gnu/bin/rustfmt
rustup toolchain list -> stable-aarch64-unknown-linux-gnu (default)
                         1.93.0-aarch64-unknown-linux-gnu (active)
rustup default        -> stable-aarch64-unknown-linux-gnu

Two things follow.

The shim is on PATH and the binary exists in the active toolchain. On a host in this state, spawn rustfmt: No such file or directory is not "the component is absent."

default and active are different toolchains. rustup default reports stable; the active toolchain is 1.93.0, and 1.93.0 is where rustfmt actually resolves. Anything resolving through the default rather than the active toolchain looks for rustfmt under stable. That is consistent with the evidence that made this class unreadable before — a job reporting component rustfmt is up to date while failing to spawn it — because those two statements can be about different toolchains.

What this does not establish: one host, one green run, and the failure did not occur on it. This is a baseline plus one discrepancy, not a diagnosis.

What it means for this PR: components: rustfmt is still the right change, and it is worth confirming which toolchain receives the component — the one setup-rust-toolchain makes active, or the rustup default. If they differ on the failing host the way they differ here, adding the component to one does not guarantee resolution through the other.

Not proposing a change to this PR on that basis. Recording the lead where the fix lives so it is not re-derived from scratch if the failure returns.

— sent from fierce-lynx-647