Skip to content

One witness per subject where a conjunction over five was the corpus's most expensive row - #9270

Merged
gunbai-bot[bot] merged 1 commit into
mainfrom
fix/root4-witness-cost
Aug 26, 2026
Merged

gunbai-bot[bot] merged 1 commit into
mainfrom
fix/root4-witness-cost

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

One witness per subject, where a conjunction over five was the most expensive row in the corpus

root4_measure_missing_generics_witness_holds conjoined five sub-witnesses, four of which each run a full compile_dag_rust_emit_check — a separate emit compile per snippet. So it cost roughly five times a one-snippet witness by construction. This splits it into five witnesses, one per subject.

The cost, from the floor's own instrument

The required floor emits one [over-cost] <identity> wall_ms= cpu_ms= line_ms= outcome= line per row over the 1552ms disclosure line. This row's cpu_ms across five consecutive main runs:

run cpu_ms vs 1552ms line vs 5000ms fail-stop
c5d52fe67 3322 2.1× under
83d7f5b5d 3901 2.5× under
9b4a3d2be 3998 2.6× under
045ab6278 4252 2.7× under
2a2804f4d 5623 3.6× over — failed the required run

Roster stable at planned=11095–11096 across all five, so the row was present in every run and nothing was missing from the denominator.

The median is the finding, not the excursion. Every one of the five is 2.1×–3.6× the disclosure line — the row was over the line in its best run. This is not a well-behaved witness that got unlucky against a tight threshold; it is an expensive row whose ordinary variance reaches the fail-stop. The gate had been reporting this debt all along.

That distinction decided the remedy. The alternative on the table was to reconsider whether completed_over_cost_requirement should block — which would have meant reversing the recorded operator ruling of 2026-08-19, to accommodate a row that should not be near the limit in the first place. Ruled against: fix the row, leave the 5000ms limit and the ruling untouched. Nothing about the gate changes here.

The split is strictly stronger than the conjunction it replaces

a && b && c && d && e reports one verdict for five subjects. When it goes red, nothing says which subject regressed, and the four that held are indistinguishable from the one that did not. Five witnesses name the failing subject in the failure itself.

Nothing is asserted less. The same five predicates run over the same five fixtures with the same required and excluded spellings. No assertion was trimmed to fit a budget — a witness made cheap by asserting less is the failure this work exists to avoid, and if the cost had turned out to be intrinsic to what the row asserts the right move would have been to stop and say so rather than shave it.

The w_ names are unchanged, so the RED-control prose in the module note — which names w_zero_param_alias_use_site_stays_bare and w_unrelated_zero_param_leaf_unchanged as its controls — still names its subjects correctly. No prose rots.

Safety of the rename

Grepped before splitting: root4_measure_missing_generics_witness_holds appears in no roster, quarantine list, or route-gap ledger — the only references anywhere are the module's own declaration and a file-path mention in docs/plans/nat-grounding-unification-design.md. Renaming a witness that is enrolled reds the gating rosters; this one is not.

Execution

All five executed standalone and returned true.

Local wall times are not quoted as floor costs and are not evidence of the improvement: a standalone gunbc run is dominated by corpus startup and runs with the memo disabled, which is 2–3× off the floor's per-claim cost. What the local runs establish is that the five witnesses are green when standalone. The cost claim rests entirely on the [over-cost] readings above, which come from the required runs themselves.

What this does not claim

This does not make the row's total work smaller — five witnesses still run four emit compiles between them. What it does is stop any single claim from carrying five subjects' cost into one 5000ms budget, and make a red name its subject. Whether the four compile-based subjects can share a resolution is a separate question I have not investigated.

Precedent: #9269, the same shape and the same argument on split_dispositions_fabricate_no_loop (four subjects, cpu 4058 in the same failing run — second on the over-cost list behind this one).

— sent from cool-hawk-324


This is a specific repair, not a precedent for splitting on shape

A census of all 150 rows on the required floor's [over-cost] list was run after this PR was opened, and it refutes the generalization that motivated looking for a second one:

  • 128 of the 150 assert exactly one subject. The two rows split (this one and its sibling) are ranks 1 and 2 by cost; the multi-subject pattern does not survive rank 3.
  • The corpus-wide median gap (multi 3372ms vs single 2198ms) is module-confounded. 15 of the 22 multi-subject rows live in test.claim.live_deploy.emit; within that module, single-subject median is 3358ms against multi-subject 3375ms — identical. Cost tracks module, not shape.
  • The sharper hypothesis — that the driver is multiple compile subjects — fails in the opposite direction: rows reaching a compile/emit check median 1890ms against 2540ms for rows that don't.
  • Corpus-wide the shape is common and cheap: 3,700 of 12,635 test fn declarations are multi-subject (29.3%), 411 with six or more subjects, and almost all of them have always been under the disclosure line.

What survives is this row on its own facts: it carried several independent subjects into one 5000ms budget, it was at or over the fail-stop, and a red named no subject. That is sufficient here and generalizes to nothing. Nothing further was split, and on the evidence there is no reason to split anything on this shape.

The better-supported lead the census produced is per-family fixture cost — live_deploy.emit is 14% of the over-cost list on its own, at a cost independent of what any individual row asserts. That is logged, not started.

— sent from cool-hawk-324

…s most expensive row

`root4_measure_missing_generics_witness_holds` conjoined five sub-witnesses, four of which
each run a full `compile_dag_rust_emit_check` -- a separate emit compile per snippet -- so it
cost roughly five times a one-snippet witness by construction. This splits it into five.

The cost, from the required floor's own `[over-cost]` instrument, across five consecutive main
runs (cpu_ms against a 1552ms disclosure line and a 5000ms fail-stop):

    c5d52fe 3322   83d7f5b 3901   9b4a3d2 3998   045ab62 4252   2a2804f 5623

Roster stable at planned=11095-11096 across all five, so the row was present in every run and
nothing was missing from the denominator. The median is the finding rather than the excursion:
every one of the five is 2.1x-3.6x the disclosure line, so the row was over the line in its
BEST run. This is not a well-behaved witness that got unlucky against a tight threshold, it is
an expensive row whose ordinary variance reaches the fail-stop, and the gate had been
reporting the debt all along.

That distinction decided the remedy. The alternative was to reconsider whether
`completed_over_cost_requirement` should block, which would have reversed the recorded
operator ruling of 2026-08-19 to accommodate a row that should not be near the limit. Ruled
against: fix the row, leave the limit and the ruling untouched. Nothing about the gate changes.

The split is strictly STRONGER than the conjunction it replaces. `a && b && c && d && e`
reports one verdict for five subjects: when it goes red, nothing says which subject regressed,
and the four that held are indistinguishable from the one that did not. Nothing is asserted
less -- the same five predicates over the same five fixtures with the same required and
excluded spellings. The `w_` names are unchanged, so the RED-control prose in the module note
still names its subjects correctly.

Grepped before splitting: the old identity appears in no roster, quarantine list or route-gap
ledger, so this cannot red a gating roster.

All five executed standalone and returned `true`. Local wall times are NOT quoted as floor
costs -- a standalone run is dominated by corpus startup and runs memo-disabled -- so the cost
claim rests entirely on the `[over-cost]` readings above, taken from the required runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAQiYQiQ7SsdxvotAFbUFc
@gunbai-bot
gunbai-bot Bot merged commit b138b0c into main Aug 26, 2026
3 checks passed
@gunbai-bot
gunbai-bot Bot deleted the fix/root4-witness-cost branch August 26, 2026 02:46
gunbai-bot Bot pushed a commit that referenced this pull request Aug 26, 2026
…des of the three conflicts

Merge of origin/main (11 commits) conflicted in witness_floor_workflow.dag and its
emitted witnesses.yml. All three hunks were BOTH-SIDES-WANTED, not either/or:

  1. imports        -- my capability widening [Cargo, Rustc, Rustfmt, Rustup]
                       AND main's new repo_self_build_all_bins_command.
  2. prepared_bound_steps -- main parameterized it as (build_script: String) for
                       #9270's two-lane bootstrap split; my capability-fork note is
                       kept beside main's two-lanes note, both at module-item grain.
  3. the step list  -- my pin-step binding AND main's
                       witness_floor_build_step(build_script: build_script).

Taking either side alone would have silently dropped the other's change, which is why
each hunk is resolved by composition rather than by choosing.

THE .yml WAS NOT HAND-RESOLVED. It is a committed generated artifact, so a hand-merged
one would be a guess wearing the authority of a generated file. Instead: reset it to
main, rebuild the toolchain binary against the MERGED tree (2m53s), and regenerate. Both
jobs carry Isolate toolchain homes and Pin rustup default, in the required order, and
main's build-lane change survives.

fleet-converge.yml again regenerated with the same unrelated secret-hygiene drift it
carries on main (8 insertions, 4 deletions, ZERO toolchain/isolate/rustup/MAKEFLAGS/
sccache lines -- measured, not assumed). Reverted again; still needs its own cut.

Capability closure re-run on the MERGED tree, RC=0, 4/4 PASS including the
fleet-converge control and the two ordering REDs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
briansrls pushed a commit that referenced this pull request Aug 26, 2026
… slots stop rewriting each other's binaries (#9275)

* Isolate the Rust toolchain homes in required CI, so concurrent runner slots stop rewriting each other's binaries

LINE STOP. The required evidence path was corrupting itself. rustfmt refused with
ETXTBSY on srv4-11 and srv3-11 seven minutes apart, killing the build lane on two
unrelated PRs, and srv1-15 resolved rustc while cargo had vanished (exit 127). The
compiler's own refusal named the cause exactly: the tool "was resolved at admission
from PATH ... and ran there: --version was executed successfully before this phase
began. So it has been removed, replaced or made unusable while this run was executing;
this is not a missing or broken installation."

CAUSE: each host runs many runner slots under one /home/ghrunner, and every witness job
began by installing rustup into it. One slot's install rewrites a toolchain another slot
is executing.

THE CONSTRUCTION ALREADY EXISTED. fleet-converge has wiped and set HOME, CARGO_HOME and
RUSTUP_HOME under RUNNER_TEMP since someone hit this before -- its pin step is literally
named "isolated RUSTUP_HOME has no default toolchain". witnesses.yml had no such step and
called setup-rust-toolchain into the shared default. So this is applying a proven in-repo
pattern, not designing a fix.

WHY A THIRD MODULE. ci_isolate_toolchain_step carried THREE concepts: home isolation,
build tuning (MAKEFLAGS, CARGO_BUILD_JOBS) and cache policy (sccache). Importing it
wholesale would have imposed a CARGO_BUILD_JOBS derived from the SELECTED runner target's
RAM-speed budget on a job with a different subject -- a build-policy change smuggled
inside an outage fix. So the intent is decomposed into ci_toolchain_home_isolation_steps
and ci_fleet_build_tuning_steps, and gunbc.toolchain_workflow_steps holds the two steps
both workflows genuinely share. The witness workflow does not import the fleet prelude
(it would borrow a cross-job artifact architecture it does not have) and does not
duplicate the shell strings (that is the §3 nicknaming violation).

THE COMPOSED FLEET PIPELINE IS BYTE-IDENTICAL BY CONSTRUCTION -- same seven steps, same
order, same builder -- and that was the discriminating control on the refactor. Verified
by execution: regenerating produced ZERO toolchain, isolate, rustup, CARGO_HOME,
MAKEFLAGS or sccache line changes in fleet-converge.yml.

THE CAPABILITY CLOSURE CAUGHT A REAL FORK, which is the part I did not predict. Adding
the pin step made the_live_witness_floor_job_closes_its_capabilities FAIL: the pin
consumes RustupCapability and the witness toolchain step declared only
[Cargo, Rustfmt], while fleet declares [Cargo, Rustc, Rustfmt, Rustup] for the IDENTICAL
setup_rust_action. One action, two capability stories. The under-declaration is corrected
rather than the consumer weakened -- the action really does install rustup and rustc --
and it went unnoticed only because nothing consumed Rustup here until now.

THE DIAGNOSTIC NOW FAILS CLOSED. It printed ls -l "${CARGO_HOME:-$HOME/.cargo}/bin",
whose :- fallback made a MISSING isolation step look like a legitimate shared-home mode.
It now refuses with ToolchainHomesNotIsolated when either home is unset, so the condition
this PR fixes cannot silently return.

YAML REGENERATED FROM THE EMITTER, not hand-edited: I rebuilt the toolchain binary (3m)
rather than hand-apply, because this change adds steps to two jobs and hand-application
at that structure is not the deterministic substitution the budget-tree.md case was.

SEPARATELY FOUND, DELIBERATELY NOT INCLUDED: .github/workflows/fleet-converge.yml is
STALE ON MAIN against its own emitter -- an unrelated secret-hygiene change (auth headers
moved from -H arguments into files) is authored but never regenerated. My heal run picked
it up; I reverted it so this PR does not absorb somebody else's change under a toolchain
title. It needs its own cut.

Witnesses: workflow_capability_closure_witness_test 4/4 PASS including the live witness
floor job, the live fleet-converge build job (the control), and the two ordering REDs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Move the capability-fork note to module-item grain: it was seven §4c violations inside a declaration body

The v2-emission phase refused with 7 hard diagnostics, all mine, all one cause:

  source annotation sits inside a declaration body. Only module-item grain is
  modeled; move it above the declaration it describes.

I put a seven-line // block INSIDE the prepared_bound_steps() list literal, beside
the binding it explains. DESIGN §4c admits only standalone leading blocks attached to
MODULE-SCOPE declarations, and each // line counts as its own annotation -- hence seven
diagnostics from one comment. Re-homed above fn prepared_bound_steps, reworded to name
the binding it now sits above rather than pointing at "below".

WHY MY LOCAL RUN WAS GREEN AND CI WAS NOT, because that is the reusable part. The
capability witnesses passed 4/4 locally through claim_batch, which evaluates and is
LENIENT about in-body annotations. The refusal lives in strict preparation / compile,
which is what CI runs. So a green witness suite is not evidence a .dag file is
admissible -- the two ask different questions, and the cheap strict check is a compile
of the entry.

Verified this time by structure rather than by rerunning the lenient path: zero indented
// lines inside any declaration body across all four files this PR touches.

WHAT THE FAILED RUN ALSO PROVED, and it is the thing the PR is for. The isolation
WORKED on a real runner. srv1-14 reported HOME, CARGO_HOME and RUSTUP_HOME all resolved
under the per-job RUNNER_TEMP, the new fail-closed diagnostic passed rather than
refusing, and the job then built for eighteen minutes through regen with no ETXTBSY and
no missing cargo -- the exact failure that killed three lanes tonight did not recur.
The rustup line "$HOME differs from euid-obtained home directory" is a non-fatal notice;
fleet-converge has produced it since it adopted the same construction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Regenerate witnesses.yml against the merged emitter, and keep both sides of the three conflicts

Merge of origin/main (11 commits) conflicted in witness_floor_workflow.dag and its
emitted witnesses.yml. All three hunks were BOTH-SIDES-WANTED, not either/or:

  1. imports        -- my capability widening [Cargo, Rustc, Rustfmt, Rustup]
                       AND main's new repo_self_build_all_bins_command.
  2. prepared_bound_steps -- main parameterized it as (build_script: String) for
                       #9270's two-lane bootstrap split; my capability-fork note is
                       kept beside main's two-lanes note, both at module-item grain.
  3. the step list  -- my pin-step binding AND main's
                       witness_floor_build_step(build_script: build_script).

Taking either side alone would have silently dropped the other's change, which is why
each hunk is resolved by composition rather than by choosing.

THE .yml WAS NOT HAND-RESOLVED. It is a committed generated artifact, so a hand-merged
one would be a guess wearing the authority of a generated file. Instead: reset it to
main, rebuild the toolchain binary against the MERGED tree (2m53s), and regenerate. Both
jobs carry Isolate toolchain homes and Pin rustup default, in the required order, and
main's build-lane change survives.

fleet-converge.yml again regenerated with the same unrelated secret-hygiene drift it
carries on main (8 insertions, 4 deletions, ZERO toolchain/isolate/rustup/MAKEFLAGS/
sccache lines -- measured, not assumed). Reverted again; still needs its own cut.

Capability closure re-run on the MERGED tree, RC=0, 4/4 PASS including the
fleet-converge control and the two ordering REDs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Move the isolation refusal into a command row beside its siblings, and mark the extracted shell carrier

Both findings from review 56091, verified against the tree before acting.

FINDING 2 IS CORRECT AND IS FIXED. The refusal was hand-built shell control flow inside
a concat chain in gunbc.witness_floor_workflow -- the medium-as-string form. The accepted
construction in this repository is a named command row composed through the pipeline, and
ci_pin_rustup_cargo_bin_missing_command is nearly identical in shape:

  if [ -z "$CARGO_BIN" ]; then echo "::error::..."; exit 1; fi

So the refusal is now ci_toolchain_homes_isolated_or_refuse_command, a data row in
v2.workflow.ci_workflow_run_emit beside its four siblings, and the probe composes it
rather than spelling shell. Emitted identically into both jobs.

FINDING 1 IS ACTIONED, WITH ITS PREMISE CORRECTED. The marker is added -- a Scaffold
disposition plus an unbound dissolution trigger, matching ci_native_cache_root_script's
construction, emitted as a leading comment in the script. But "the new shell carrier" is
not what ci_toolchain_home_isolation_script is: it authors NO new shell. It emits four
PRE-EXISTING command rows unchanged (ci_isolate_rm_rustup_cargo_command and the three
echo rows) that ci_isolate_toolchain_script already emitted. What this PR added was a
COMPOSITION, not a medium.

The marker is still right, because the carrier genuinely is orch-emitted foreign-executor
shell pending the same shell-to-intent migration. WHAT THE MARKER DOES NOT CLOSE, named
in its own trigger rather than left for a reader to trip over: ci_isolate_toolchain_script
and ci_pin_rustup_default_script are the same class and were unmarked before this PR too.
Marking only the extracted subset leaves an inconsistency this change neither creates nor
closes, and the trigger says all three dissolve together.

ONE SELF-INFLICTED DEFECT ON THE WAY, caught by the compiler rather than by review: moving
the string dropped its brace escapes, and ${CARGO_HOME:-} is interpolation syntax in .dag.
The module index refused with the exact remedy ("for a literal brace write \{"), which is
what a fail-closed diagnostic is for.

Regenerated from the emitter against a rebuilt binary. fleet-converge.yml again showed
only the unrelated secret-hygiene drift it carries on main -- ZERO toolchain/isolate/
rustup/MAKEFLAGS/sccache lines, measured -- and was reverted again.

Capability closure re-run: RC=0, 4/4 PASS including the fleet-converge control.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Close the fleet-desired shared-home gap, and derive the toolchain-home standing from the emitted job

fleet-desired.yml installed a Rust toolchain into /home/ghrunner on srv1, where many runner
slots share one home -- the same defect witnesses.yml carried, with the remedy sitting one
import away in gunbc.toolchain_workflow_steps the whole time.

The job now composes toolchain_home_isolation_step ahead of the install and
toolchain_pin_rustup_default_step behind it, in witness-floor's own order.

gunbc.toolchain_home_standing carries the obligation rather than the remedy: it folds
Workflow.jobs and classifies each job PrivateMutableHome or ImmutableToolchain off its emitted
steps -- against the authority's own command rows and the cited setup_rust_action, never a text
pattern -- and refuses on a shared-home install, an isolation with no install, or an
undecidable expression runner. The predicate is conjoined into all three emitters, so a job
added without a standing emits no yaml at all.

action_ref_eq moves from gunbc.ci_heal_credential to extdeps.github.actions: it is a fact about
ActionRef, and a second consumer would have made a credential-healing module the authority on
action identity.

* Regenerate .gitattributes: main landed partition_crate_boundary_host.rs without its merge-attribute row

Not this branch's change. #9284 added the stage0 host file and the artifact is behind its
authority by exactly the one derived row that file entails, so every branch's regen gate
reproduces it. Carried here because the required run regenerates before it compares.

* The pin script is a different removable unit from the isolation carrier, so it owes its own obligation

review 56168 is right on the doctrine. docs/plans/scaffold-admission-doctrine.md admits a
SHARED dissolution obligation only where the artifacts have the same terminal construction,
fire on the same observation, AND CANNOT SURVIVE ONE ANOTHER. ci_pin_rustup_default_script
fails the first and third: it selects a TOOLCHAIN through rustup, which is not the per-job
filesystem-and-environment effect ci_toolchain_home_isolation_script waits on, so when that
effect lands the isolation shell dissolves and the pin shell is still standing.

The previous trigger swept it in as a sibling that "dissolves together". That is precisely the
single broad trigger laundering an expanding collection of temporary work the doctrine names,
so the claim is WITHDRAWN rather than reworded, and the pin script now carries its own Scaffold
disposition and its own unbound_dissolution naming the toolchain-selection effect it actually
waits on.

ZERO EXECUTABLE SHELL MOVED, and that is checked rather than asserted: filtering the emitted
workflow diff to lines that are not dissolve-on comments returns EMPTY across all three
artifacts. The change is four added comment lines and one corrected comment line.

ONE CLAIM IN THIS PR IS NARROWED BY IT. The branch said fleet-converge.yml does not move a
byte; it now moves by exactly one line, the pin step's own marker. The step's NAME and its
executable body are still unchanged, which is what the byte-identity claim existed to protect
-- the cost-floor exempt roster keys on step name against the live workflow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Replace the Bool admission with a typed verdict, and mark the pin carrier on its own declaration

review 56168, both findings.

The Bool predicate is deleted rather than given a disposition receipt. std.keyed_roster's own
carrier already rules against it in one sentence -- consumers inspect the canonical typed
verdict directly, with no parallel *_holds predicate that collapses the coproduct -- and the
collapse was not free here: it discarded the job_id and the ToolchainHomeRefusal at the exact
seam the emitter needed them, which is why all three emitters named one static string.
admit_workflow_toolchain_homes returns WorkflowToolchainHomeAdmission, the refusing arm
reproduces itself through the fold, and each cause now renders its own remedy -- the three the
ToolchainHomeRefusal note promised and the emission was not keeping. WitnessFloorGenerationRefused
and FleetDesiredGenerationRefused gained a reason field to carry it; a toolchain-home refusal and
a capability-closure refusal previously reached the operator as the same sentence.

The witness keeps a Bool, because a witness returns Bool by construction -- the collapse belongs
at the assertion, where the discarded fields have no remaining consumer.

ci_pin_rustup_default_script carries its own Scaffold and DissolutionCondition. Being named in
the isolation carrier's trigger text bound nothing removable, and this PR triples the step's
reach. ci_isolate_toolchain_script stays unmarked and the declaration says so: this PR does not
touch it, and marking it would put a comment line into an emitted script no finding is about.

Emitted yaml changes by marker comment only; no semantic delta.

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants