Repository navigation
One witness per subject where a conjunction over five was the corpus's most expensive row - #9270
Merged
Merged
Conversation
…s most expensive row
`root4_measure_missing_generics_witness_holds` conjoined five sub-witnesses, four of which
each run a full `compile_dag_rust_emit_check` -- a separate emit compile per snippet -- so it
cost roughly five times a one-snippet witness by construction. This splits it into five.
The cost, from the required floor's own `[over-cost]` instrument, across five consecutive main
runs (cpu_ms against a 1552ms disclosure line and a 5000ms fail-stop):
c5d52fe 3322 83d7f5b 3901 9b4a3d2 3998 045ab62 4252 2a2804f 5623
Roster stable at planned=11095-11096 across all five, so the row was present in every run and
nothing was missing from the denominator. The median is the finding rather than the excursion:
every one of the five is 2.1x-3.6x the disclosure line, so the row was over the line in its
BEST run. This is not a well-behaved witness that got unlucky against a tight threshold, it is
an expensive row whose ordinary variance reaches the fail-stop, and the gate had been
reporting the debt all along.
That distinction decided the remedy. The alternative was to reconsider whether
`completed_over_cost_requirement` should block, which would have reversed the recorded
operator ruling of 2026-08-19 to accommodate a row that should not be near the limit. Ruled
against: fix the row, leave the limit and the ruling untouched. Nothing about the gate changes.
The split is strictly STRONGER than the conjunction it replaces. `a && b && c && d && e`
reports one verdict for five subjects: when it goes red, nothing says which subject regressed,
and the four that held are indistinguishable from the one that did not. Nothing is asserted
less -- the same five predicates over the same five fixtures with the same required and
excluded spellings. The `w_` names are unchanged, so the RED-control prose in the module note
still names its subjects correctly.
Grepped before splitting: the old identity appears in no roster, quarantine list or route-gap
ledger, so this cannot red a gating roster.
All five executed standalone and returned `true`. Local wall times are NOT quoted as floor
costs -- a standalone run is dominated by corpus startup and runs memo-disabled -- so the cost
claim rests entirely on the `[over-cost]` readings above, taken from the required runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAQiYQiQ7SsdxvotAFbUFc
gunbai-bot Bot
pushed a commit
that referenced
this pull request
Aug 26, 2026
…des of the three conflicts
Merge of origin/main (11 commits) conflicted in witness_floor_workflow.dag and its
emitted witnesses.yml. All three hunks were BOTH-SIDES-WANTED, not either/or:
1. imports -- my capability widening [Cargo, Rustc, Rustfmt, Rustup]
AND main's new repo_self_build_all_bins_command.
2. prepared_bound_steps -- main parameterized it as (build_script: String) for
#9270's two-lane bootstrap split; my capability-fork note is
kept beside main's two-lanes note, both at module-item grain.
3. the step list -- my pin-step binding AND main's
witness_floor_build_step(build_script: build_script).
Taking either side alone would have silently dropped the other's change, which is why
each hunk is resolved by composition rather than by choosing.
THE .yml WAS NOT HAND-RESOLVED. It is a committed generated artifact, so a hand-merged
one would be a guess wearing the authority of a generated file. Instead: reset it to
main, rebuild the toolchain binary against the MERGED tree (2m53s), and regenerate. Both
jobs carry Isolate toolchain homes and Pin rustup default, in the required order, and
main's build-lane change survives.
fleet-converge.yml again regenerated with the same unrelated secret-hygiene drift it
carries on main (8 insertions, 4 deletions, ZERO toolchain/isolate/rustup/MAKEFLAGS/
sccache lines -- measured, not assumed). Reverted again; still needs its own cut.
Capability closure re-run on the MERGED tree, RC=0, 4/4 PASS including the
fleet-converge control and the two ordering REDs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
briansrls
pushed a commit
that referenced
this pull request
Aug 26, 2026
… slots stop rewriting each other's binaries (#9275) * Isolate the Rust toolchain homes in required CI, so concurrent runner slots stop rewriting each other's binaries LINE STOP. The required evidence path was corrupting itself. rustfmt refused with ETXTBSY on srv4-11 and srv3-11 seven minutes apart, killing the build lane on two unrelated PRs, and srv1-15 resolved rustc while cargo had vanished (exit 127). The compiler's own refusal named the cause exactly: the tool "was resolved at admission from PATH ... and ran there: --version was executed successfully before this phase began. So it has been removed, replaced or made unusable while this run was executing; this is not a missing or broken installation." CAUSE: each host runs many runner slots under one /home/ghrunner, and every witness job began by installing rustup into it. One slot's install rewrites a toolchain another slot is executing. THE CONSTRUCTION ALREADY EXISTED. fleet-converge has wiped and set HOME, CARGO_HOME and RUSTUP_HOME under RUNNER_TEMP since someone hit this before -- its pin step is literally named "isolated RUSTUP_HOME has no default toolchain". witnesses.yml had no such step and called setup-rust-toolchain into the shared default. So this is applying a proven in-repo pattern, not designing a fix. WHY A THIRD MODULE. ci_isolate_toolchain_step carried THREE concepts: home isolation, build tuning (MAKEFLAGS, CARGO_BUILD_JOBS) and cache policy (sccache). Importing it wholesale would have imposed a CARGO_BUILD_JOBS derived from the SELECTED runner target's RAM-speed budget on a job with a different subject -- a build-policy change smuggled inside an outage fix. So the intent is decomposed into ci_toolchain_home_isolation_steps and ci_fleet_build_tuning_steps, and gunbc.toolchain_workflow_steps holds the two steps both workflows genuinely share. The witness workflow does not import the fleet prelude (it would borrow a cross-job artifact architecture it does not have) and does not duplicate the shell strings (that is the §3 nicknaming violation). THE COMPOSED FLEET PIPELINE IS BYTE-IDENTICAL BY CONSTRUCTION -- same seven steps, same order, same builder -- and that was the discriminating control on the refactor. Verified by execution: regenerating produced ZERO toolchain, isolate, rustup, CARGO_HOME, MAKEFLAGS or sccache line changes in fleet-converge.yml. THE CAPABILITY CLOSURE CAUGHT A REAL FORK, which is the part I did not predict. Adding the pin step made the_live_witness_floor_job_closes_its_capabilities FAIL: the pin consumes RustupCapability and the witness toolchain step declared only [Cargo, Rustfmt], while fleet declares [Cargo, Rustc, Rustfmt, Rustup] for the IDENTICAL setup_rust_action. One action, two capability stories. The under-declaration is corrected rather than the consumer weakened -- the action really does install rustup and rustc -- and it went unnoticed only because nothing consumed Rustup here until now. THE DIAGNOSTIC NOW FAILS CLOSED. It printed ls -l "${CARGO_HOME:-$HOME/.cargo}/bin", whose :- fallback made a MISSING isolation step look like a legitimate shared-home mode. It now refuses with ToolchainHomesNotIsolated when either home is unset, so the condition this PR fixes cannot silently return. YAML REGENERATED FROM THE EMITTER, not hand-edited: I rebuilt the toolchain binary (3m) rather than hand-apply, because this change adds steps to two jobs and hand-application at that structure is not the deterministic substitution the budget-tree.md case was. SEPARATELY FOUND, DELIBERATELY NOT INCLUDED: .github/workflows/fleet-converge.yml is STALE ON MAIN against its own emitter -- an unrelated secret-hygiene change (auth headers moved from -H arguments into files) is authored but never regenerated. My heal run picked it up; I reverted it so this PR does not absorb somebody else's change under a toolchain title. It needs its own cut. Witnesses: workflow_capability_closure_witness_test 4/4 PASS including the live witness floor job, the live fleet-converge build job (the control), and the two ordering REDs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move the capability-fork note to module-item grain: it was seven §4c violations inside a declaration body The v2-emission phase refused with 7 hard diagnostics, all mine, all one cause: source annotation sits inside a declaration body. Only module-item grain is modeled; move it above the declaration it describes. I put a seven-line // block INSIDE the prepared_bound_steps() list literal, beside the binding it explains. DESIGN §4c admits only standalone leading blocks attached to MODULE-SCOPE declarations, and each // line counts as its own annotation -- hence seven diagnostics from one comment. Re-homed above fn prepared_bound_steps, reworded to name the binding it now sits above rather than pointing at "below". WHY MY LOCAL RUN WAS GREEN AND CI WAS NOT, because that is the reusable part. The capability witnesses passed 4/4 locally through claim_batch, which evaluates and is LENIENT about in-body annotations. The refusal lives in strict preparation / compile, which is what CI runs. So a green witness suite is not evidence a .dag file is admissible -- the two ask different questions, and the cheap strict check is a compile of the entry. Verified this time by structure rather than by rerunning the lenient path: zero indented // lines inside any declaration body across all four files this PR touches. WHAT THE FAILED RUN ALSO PROVED, and it is the thing the PR is for. The isolation WORKED on a real runner. srv1-14 reported HOME, CARGO_HOME and RUSTUP_HOME all resolved under the per-job RUNNER_TEMP, the new fail-closed diagnostic passed rather than refusing, and the job then built for eighteen minutes through regen with no ETXTBSY and no missing cargo -- the exact failure that killed three lanes tonight did not recur. The rustup line "$HOME differs from euid-obtained home directory" is a non-fatal notice; fleet-converge has produced it since it adopted the same construction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Regenerate witnesses.yml against the merged emitter, and keep both sides of the three conflicts Merge of origin/main (11 commits) conflicted in witness_floor_workflow.dag and its emitted witnesses.yml. All three hunks were BOTH-SIDES-WANTED, not either/or: 1. imports -- my capability widening [Cargo, Rustc, Rustfmt, Rustup] AND main's new repo_self_build_all_bins_command. 2. prepared_bound_steps -- main parameterized it as (build_script: String) for #9270's two-lane bootstrap split; my capability-fork note is kept beside main's two-lanes note, both at module-item grain. 3. the step list -- my pin-step binding AND main's witness_floor_build_step(build_script: build_script). Taking either side alone would have silently dropped the other's change, which is why each hunk is resolved by composition rather than by choosing. THE .yml WAS NOT HAND-RESOLVED. It is a committed generated artifact, so a hand-merged one would be a guess wearing the authority of a generated file. Instead: reset it to main, rebuild the toolchain binary against the MERGED tree (2m53s), and regenerate. Both jobs carry Isolate toolchain homes and Pin rustup default, in the required order, and main's build-lane change survives. fleet-converge.yml again regenerated with the same unrelated secret-hygiene drift it carries on main (8 insertions, 4 deletions, ZERO toolchain/isolate/rustup/MAKEFLAGS/ sccache lines -- measured, not assumed). Reverted again; still needs its own cut. Capability closure re-run on the MERGED tree, RC=0, 4/4 PASS including the fleet-converge control and the two ordering REDs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move the isolation refusal into a command row beside its siblings, and mark the extracted shell carrier Both findings from review 56091, verified against the tree before acting. FINDING 2 IS CORRECT AND IS FIXED. The refusal was hand-built shell control flow inside a concat chain in gunbc.witness_floor_workflow -- the medium-as-string form. The accepted construction in this repository is a named command row composed through the pipeline, and ci_pin_rustup_cargo_bin_missing_command is nearly identical in shape: if [ -z "$CARGO_BIN" ]; then echo "::error::..."; exit 1; fi So the refusal is now ci_toolchain_homes_isolated_or_refuse_command, a data row in v2.workflow.ci_workflow_run_emit beside its four siblings, and the probe composes it rather than spelling shell. Emitted identically into both jobs. FINDING 1 IS ACTIONED, WITH ITS PREMISE CORRECTED. The marker is added -- a Scaffold disposition plus an unbound dissolution trigger, matching ci_native_cache_root_script's construction, emitted as a leading comment in the script. But "the new shell carrier" is not what ci_toolchain_home_isolation_script is: it authors NO new shell. It emits four PRE-EXISTING command rows unchanged (ci_isolate_rm_rustup_cargo_command and the three echo rows) that ci_isolate_toolchain_script already emitted. What this PR added was a COMPOSITION, not a medium. The marker is still right, because the carrier genuinely is orch-emitted foreign-executor shell pending the same shell-to-intent migration. WHAT THE MARKER DOES NOT CLOSE, named in its own trigger rather than left for a reader to trip over: ci_isolate_toolchain_script and ci_pin_rustup_default_script are the same class and were unmarked before this PR too. Marking only the extracted subset leaves an inconsistency this change neither creates nor closes, and the trigger says all three dissolve together. ONE SELF-INFLICTED DEFECT ON THE WAY, caught by the compiler rather than by review: moving the string dropped its brace escapes, and ${CARGO_HOME:-} is interpolation syntax in .dag. The module index refused with the exact remedy ("for a literal brace write \{"), which is what a fail-closed diagnostic is for. Regenerated from the emitter against a rebuilt binary. fleet-converge.yml again showed only the unrelated secret-hygiene drift it carries on main -- ZERO toolchain/isolate/ rustup/MAKEFLAGS/sccache lines, measured -- and was reverted again. Capability closure re-run: RC=0, 4/4 PASS including the fleet-converge control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Close the fleet-desired shared-home gap, and derive the toolchain-home standing from the emitted job fleet-desired.yml installed a Rust toolchain into /home/ghrunner on srv1, where many runner slots share one home -- the same defect witnesses.yml carried, with the remedy sitting one import away in gunbc.toolchain_workflow_steps the whole time. The job now composes toolchain_home_isolation_step ahead of the install and toolchain_pin_rustup_default_step behind it, in witness-floor's own order. gunbc.toolchain_home_standing carries the obligation rather than the remedy: it folds Workflow.jobs and classifies each job PrivateMutableHome or ImmutableToolchain off its emitted steps -- against the authority's own command rows and the cited setup_rust_action, never a text pattern -- and refuses on a shared-home install, an isolation with no install, or an undecidable expression runner. The predicate is conjoined into all three emitters, so a job added without a standing emits no yaml at all. action_ref_eq moves from gunbc.ci_heal_credential to extdeps.github.actions: it is a fact about ActionRef, and a second consumer would have made a credential-healing module the authority on action identity. * Regenerate .gitattributes: main landed partition_crate_boundary_host.rs without its merge-attribute row Not this branch's change. #9284 added the stage0 host file and the artifact is behind its authority by exactly the one derived row that file entails, so every branch's regen gate reproduces it. Carried here because the required run regenerates before it compares. * The pin script is a different removable unit from the isolation carrier, so it owes its own obligation review 56168 is right on the doctrine. docs/plans/scaffold-admission-doctrine.md admits a SHARED dissolution obligation only where the artifacts have the same terminal construction, fire on the same observation, AND CANNOT SURVIVE ONE ANOTHER. ci_pin_rustup_default_script fails the first and third: it selects a TOOLCHAIN through rustup, which is not the per-job filesystem-and-environment effect ci_toolchain_home_isolation_script waits on, so when that effect lands the isolation shell dissolves and the pin shell is still standing. The previous trigger swept it in as a sibling that "dissolves together". That is precisely the single broad trigger laundering an expanding collection of temporary work the doctrine names, so the claim is WITHDRAWN rather than reworded, and the pin script now carries its own Scaffold disposition and its own unbound_dissolution naming the toolchain-selection effect it actually waits on. ZERO EXECUTABLE SHELL MOVED, and that is checked rather than asserted: filtering the emitted workflow diff to lines that are not dissolve-on comments returns EMPTY across all three artifacts. The change is four added comment lines and one corrected comment line. ONE CLAIM IN THIS PR IS NARROWED BY IT. The branch said fleet-converge.yml does not move a byte; it now moves by exactly one line, the pin step's own marker. The step's NAME and its executable body are still unchanged, which is what the byte-identity claim existed to protect -- the cost-floor exempt roster keys on step name against the live workflow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Replace the Bool admission with a typed verdict, and mark the pin carrier on its own declaration review 56168, both findings. The Bool predicate is deleted rather than given a disposition receipt. std.keyed_roster's own carrier already rules against it in one sentence -- consumers inspect the canonical typed verdict directly, with no parallel *_holds predicate that collapses the coproduct -- and the collapse was not free here: it discarded the job_id and the ToolchainHomeRefusal at the exact seam the emitter needed them, which is why all three emitters named one static string. admit_workflow_toolchain_homes returns WorkflowToolchainHomeAdmission, the refusing arm reproduces itself through the fold, and each cause now renders its own remedy -- the three the ToolchainHomeRefusal note promised and the emission was not keeping. WitnessFloorGenerationRefused and FleetDesiredGenerationRefused gained a reason field to carry it; a toolchain-home refusal and a capability-closure refusal previously reached the operator as the same sentence. The witness keeps a Bool, because a witness returns Bool by construction -- the collapse belongs at the assertion, where the discarded fields have no remaining consumer. ci_pin_rustup_default_script carries its own Scaffold and DissolutionCondition. Being named in the isolation carrier's trigger text bound nothing removable, and this PR triples the step's reach. ci_isolate_toolchain_script stays unmarked and the declaration says so: this PR does not touch it, and marking it would put a comment line into an emitted script no finding is about. Emitted yaml changes by marker comment only; no semantic delta. --------- Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One witness per subject, where a conjunction over five was the most expensive row in the corpus
root4_measure_missing_generics_witness_holdsconjoined five sub-witnesses, four of which each run a fullcompile_dag_rust_emit_check— a separate emit compile per snippet. So it cost roughly five times a one-snippet witness by construction. This splits it into five witnesses, one per subject.The cost, from the floor's own instrument
The required floor emits one
[over-cost] <identity> wall_ms= cpu_ms= line_ms= outcome=line per row over the 1552ms disclosure line. This row'scpu_msacross five consecutive main runs:c5d52fe6783d7f5b5d9b4a3d2be045ab62782a2804f4dRoster stable at
planned=11095–11096across all five, so the row was present in every run and nothing was missing from the denominator.The median is the finding, not the excursion. Every one of the five is 2.1×–3.6× the disclosure line — the row was over the line in its best run. This is not a well-behaved witness that got unlucky against a tight threshold; it is an expensive row whose ordinary variance reaches the fail-stop. The gate had been reporting this debt all along.
That distinction decided the remedy. The alternative on the table was to reconsider whether
completed_over_cost_requirementshould block — which would have meant reversing the recorded operator ruling of 2026-08-19, to accommodate a row that should not be near the limit in the first place. Ruled against: fix the row, leave the 5000ms limit and the ruling untouched. Nothing about the gate changes here.The split is strictly stronger than the conjunction it replaces
a && b && c && d && ereports one verdict for five subjects. When it goes red, nothing says which subject regressed, and the four that held are indistinguishable from the one that did not. Five witnesses name the failing subject in the failure itself.Nothing is asserted less. The same five predicates run over the same five fixtures with the same required and excluded spellings. No assertion was trimmed to fit a budget — a witness made cheap by asserting less is the failure this work exists to avoid, and if the cost had turned out to be intrinsic to what the row asserts the right move would have been to stop and say so rather than shave it.
The
w_names are unchanged, so the RED-control prose in the module note — which namesw_zero_param_alias_use_site_stays_bareandw_unrelated_zero_param_leaf_unchangedas its controls — still names its subjects correctly. No prose rots.Safety of the rename
Grepped before splitting:
root4_measure_missing_generics_witness_holdsappears in no roster, quarantine list, or route-gap ledger — the only references anywhere are the module's own declaration and a file-path mention indocs/plans/nat-grounding-unification-design.md. Renaming a witness that is enrolled reds the gating rosters; this one is not.Execution
All five executed standalone and returned
true.Local wall times are not quoted as floor costs and are not evidence of the improvement: a standalone
gunbc runis dominated by corpus startup and runs with the memo disabled, which is 2–3× off the floor's per-claim cost. What the local runs establish is that the five witnesses are green when standalone. The cost claim rests entirely on the[over-cost]readings above, which come from the required runs themselves.What this does not claim
This does not make the row's total work smaller — five witnesses still run four emit compiles between them. What it does is stop any single claim from carrying five subjects' cost into one 5000ms budget, and make a red name its subject. Whether the four compile-based subjects can share a resolution is a separate question I have not investigated.
Precedent: #9269, the same shape and the same argument on
split_dispositions_fabricate_no_loop(four subjects, cpu 4058 in the same failing run — second on the over-cost list behind this one).— sent from cool-hawk-324
This is a specific repair, not a precedent for splitting on shape
A census of all 150 rows on the required floor's
[over-cost]list was run after this PR was opened, and it refutes the generalization that motivated looking for a second one:test.claim.live_deploy.emit; within that module, single-subject median is 3358ms against multi-subject 3375ms — identical. Cost tracks module, not shape.test fndeclarations are multi-subject (29.3%), 411 with six or more subjects, and almost all of them have always been under the disclosure line.What survives is this row on its own facts: it carried several independent subjects into one 5000ms budget, it was at or over the fail-stop, and a red named no subject. That is sufficient here and generalizes to nothing. Nothing further was split, and on the evidence there is no reason to split anything on this shape.
The better-supported lead the census produced is per-family fixture cost —
live_deploy.emitis 14% of the over-cost list on its own, at a cost independent of what any individual row asserts. That is logged, not started.— sent from cool-hawk-324