Stop rust-cache deleting other jobs' files: every slot shares /home/ghrunner/.cargo

[gunbai-bot]

Fixes the host tool unresolved: "cargo" failures (9 on a recent run) and the could not parse/generate dep info build failures. They are one bug.

The mechanism, confirmed against upstream source

setup-rust-toolchain defaults cache to true, which runs Swatinem/rust-cache, which registers a POST step. Two of its cleanup functions are the deleters:

function	removes
cleanBin(oldBins)	binaries in ~/.cargo/bin not in the baseline captured when that job started
cleanRegistry(…)	directories in ~/.cargo/registry/src not matching the current workspace dependency set

Both are correct on a GitHub-hosted runner, where the home belongs to one job and dies with it. This fleet is self-hosted: /opt/actions-runner/<slot>/_work is per-slot, but /home/ghrunner/.cargo is one directory shared by every slot on the host. Each job computes its baseline and dependency set as though it owned the home, then its post step deletes against a home other jobs are actively building in.

The two symptoms map one-to-one

cleanBin → cargo unresolved. Run 32088114155, one job:

01:24:48 -> 01:24:52  Install Rust toolchain   success
01:24:52 -> 01:27:41  Build the witness fold   success   <- ran cargo 2m49s
01:27:41 -> 02:13:10  All witnesses            FAILURE   <- cargo unresolved

Same job, same PATH, cargo present then absent. A directory missing from PATH cannot be present at 01:24 and absent at 01:40 — which is what rules out the configuration explanation, and the PATH fix that follows from it.

cleanRegistry → missing sources mid-compile. Run 32087371268, srv2-04:

ring-0.17.14/crypto/fipsmodule/ec/ecp_nistz.c: No such file or directory

Its downstream noise is the could not parse/generate dep info at: .../target/... class, which reads like a target/ problem and is not — the real error is the line above it, and the missing file is under ~/.cargo.

Selection effect corroborates. The witness step is ~45 minutes and the build step ~3, so nearly every hit lands in the witness step. That is what an overlap-probability argument predicts and what a configuration error would not.

Why disabling rather than splitting the home

Per-slot CARGO_HOME/RUSTUP_HOME also works and was my first candidate. It multiplies the registry by the slot count on every host, to work around a cache layer this fleet does not need: ~/.cargo already persists across jobs on a self-hosted runner, so restore-and-save buys nothing and only the cleanup half has any effect. Disabling removes the mechanism outright instead of giving it a private place to run.

The cost, stated

Nothing prunes ~/.cargo now, so the registry grows unbounded and becomes a disk-pressure question instead of a correctness one. I think that is the right trade — a full disk refuses loudly, a source file deleted mid-compile does not — but it is a real obligation and belongs to whoever owns fleet disk.

Regeneration

The yml is regenerated from dag/gunbc/witness_floor_workflow.dag, not hand-edited. A gunbc built from this tree reproduces the committed file byte-for-byte apart from the two added lines. Worth noting because no gate verifies this file against its authority — the drift gates went with the floor cut.

Independent of #8412; either can land first.

[gunbai-bot]

CI red here is not this diff, and the run carries three receipts that the fix is doing exactly what it claims.

The deleter is disabled, and you can see it in the step timings

Run 32093215764:

02:49:22 -> 02:49:24  Install Rust toolchain        success
02:49:24 -> 02:52:01  Build the witness fold        success   (2m37s)
02:52:01 -> 03:31:06  All witnesses                 FAILURE
03:31:06 -> 03:31:06  Post Install Rust toolchain   success   (0s)

Post Install Rust toolchain took 0 seconds. On a run with caching enabled (32088114155) the same step took 5 seconds. That step is Swatinem/rust-cache post, and 5s to 0s is cleanBin/cleanRegistry no longer executing. This is the mechanism being removed, observed rather than argued.

The build did not get slower: 2m37s here against 2m49s with the cache on. That confirms the reasoning in the commit — ~/.cargo already persists across jobs on a self-hosted runner, so restore-and-save was buying nothing and only the cleanup half had any effect.

cargo-unresolved failures went 9 to 4, and the remainder is predicted

	before	after
host tool unresolved: "cargo"	9	4
emit_host_transport_wire	3	0

Not zero, and that is expected rather than disappointing. This change only disables the cleanup in runs that use this workflow revision. Every other in-flight run — PR branches and main, all still on the old workflow with cache: true — is still executing cleanBin and cleanRegistry against the shared /home/ghrunner/.cargo while this job builds. So the fix protects the fleet only once it is on main and every run inherits it.

That gives a falsifiable prediction, which I would rather state now than claim afterwards: once this merges, cargo-unresolved should go to zero. If it does not, the shared-home story is incomplete and I want to know that.

Nothing failing is attributable to this diff

planned=9418 executed=9418 terminal=9418 passed=8594 failed=37 (main was 38 at a slightly older tree).

This PR touches .github/workflows/witnesses.yml and dag/gunbc/witness_floor_workflow.dag. No witness module reads either — witness_floor_workflow and expected_witness_floor_yml appear in no witness, and WitnessFloorYamlArtifact only in the two authority modules. The 37 are the standing set: 16 compute_board_admission pattern-match failures, 7 ci_heal_skew_guard budget overruns, 5 roadmap_static_site roster rows that now pass, and the 4 cargo failures above.

Two of those classes move run to run purely with host load — ci_heal_skew_guard was 4 on the previous run and is 7 here, with no code change between them, because those rows sit within milliseconds of the 500ms line. That instability is its own problem and is owned separately.

— sent from tidy-lark-471