Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/heal.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ jobs:
run: |
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
"$ROOT/target/release/gunbc" run --source-root "$ROOT/dag" --source-root "$ROOT/src/v2" --entry dag/gunbc/instruments/generated_artifact_gate.dag --function heal_repair_declaration
- name: Regenerate every committed generated artifact
- name: Regenerate every registry-rostered generated artifact (not the stage0 mirrors)
run: |
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
"$ROOT/target/release/gunbc" run --source-root "$ROOT/dag" --source-root "$ROOT/src/v2" --entry dag/gunbc/instruments/generated_artifact_gate.dag --function main_wet
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/witnesses.yml
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ jobs:
env:
REQUIRED_CI_INSTRUMENT_BUILD_OUTCOME: ${{ steps.build_witness_fold.outcome }}
if: always() && !cancelled()
- name: Generated artifacts match their authorities
- name: Registry-rostered generated artifacts (gunbc.generated_artifact) match their authorities; the stage0 mirrors are NOT checked here
id: generated_artifact_gate
run: |
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
Expand All @@ -287,7 +287,7 @@ jobs:
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
"$ROOT/target/release/gunbc" run --source-root "$ROOT/dag" --source-root "$ROOT/src/v2" --entry dag/gunbc/instruments/generated_artifact_gate.dag --function heal_repair_declaration
if: failure() && steps.generated_artifact_gate.outcome == 'failure' && github.actor != 'dependabot[bot]'
- name: Regenerate every committed generated artifact, then verify the tree it wrote
- name: Regenerate every registry-rostered generated artifact (not the stage0 mirrors), then verify the tree it wrote
id: repair_regen
run: |
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
Expand Down
4 changes: 2 additions & 2 deletions dag/gunbc/heal_workflow.dag
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ fn heal_repair_declaration_step() -> Step {

fn heal_regen_step() -> Step {
RunStep {
name: Present { value: "Regenerate every committed generated artifact" },
name: Present { value: "Regenerate every registry-rostered generated artifact (not the stage0 mirrors)" },
id: none,
run: gunbc_ci_heal_regen_invoke(),
shell: none,
Expand Down Expand Up @@ -270,7 +270,7 @@ fn heal_workflow_bound_steps() -> List<HealBoundStep> {
HealBoundStep {
step: heal_regen_step(),
role: capability_neutral,
step_name: "Regenerate every committed generated artifact",
step_name: "Regenerate every registry-rostered generated artifact (not the stage0 mirrors)",
},
HealBoundStep {
step: heal_candidate_step(),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
module gunbc.recurring_failure_mode.generated_population_split_across_two_unjoined_rosters

import std.types { NonEmptyStr }
import std.decl_ref { DeclarationRef, WholeDeclaration }
import gunbc.recurring_failure_mode { RecurringFailureMode }

data generated_population_split_across_two_unjoined_rosters: RecurringFailureMode = RecurringFailureMode {
identity: "generated_population_split_across_two_unjoined_rosters" as NonEmptyStr,

receipts: [
"the answer to 'which committed files are generated, and by what' is split across two rosters that do not know about each other, so a drift gate over ONE of them stays green over the other and its green reads as a statement about both (INVALID STATE: a committed file emitted from a .dag authority belongs to a population no required step adjudicates, while a sibling file in the same directory, emitted the same way, is adjudicated; HARM: generated bytes drift on main with every required lane green)",

"THE MEMBERSHIP TEST IS INVISIBLE FROM THE FILENAME. Under src/v1/stage0/src, bootstrap_stage0_crate_layout_generated.rs and its *_generated.rs siblings are rows of gunbc.generated_artifact committed_generated_artifacts and are checked by the required drift step; std_measure.rs, v1_rt.rs and the rest of the compile_stage0 emitted population are mirrors of .dag authorities too, but they belong ONLY to the stage0 emitted population, which claim_executor's GeneratedArtifact phase adjudicates. Both kinds sit in one directory and nothing a reader can see says which roster owns a file. That is DESIGN section 3's fork: two authorities for one fact, neither derived from the other.",

"PARTIAL COVERAGE THAT READS AS TOTAL IS THE HARM, NOT THE MISSING FILE. The GeneratedArtifact phase routes to --required-lane build, and no job has invoked that lane since #11742 (a declared drop, gunbc.rung_drop.witness_floor_off_the_required_gate). The step that DID run was named 'Generated artifacts match their authorities', so the required run stayed green while the stage0 mirrors were unchecked, and green over a step with that name reads as coverage of stage0. The drop row was honest; the step name was the inflation.",

"RECEIPT, gunbc main at aa5716f1fd3: #11962 declared std.measure kibibyte_from_byte_size_floor (consumed by gunbc.compute.host_capacity) without regenerating the mirror. #12029 regenerated src/v1/stage0/src/std_measure.rs on a base WITHOUT #11962, then merged main; the mirror had changed on one side only, so git merged it with no conflict and the stale bytes landed. The emitter was right for every tree it ran on. Instrument: claim_executor --required-regen --source-root dag --source-root src/v2 on a clean aa5716f1fd3 tree, with the binary built from that tree, reported planned=157 executed=157 and exactly one drift, std_measure.rs. Two earlier remote runs were discarded as instrument failures (HostBudgetUnreadable before any write; a runner with 7.1 GiB refused a phase peaking near 13 GiB). Repaired by gunbc#12067.",

"DISCRIMINATOR: does a single roster enumerate every committed file emitted from a .dag authority, and does every drift step read that roster? Two rosters with a hand-kept boundary between them is red even when both are currently clean, because the boundary is exactly where a later change drops a file. A step whose name claims a population larger than the roster it reads is red on its own.",

"RUNG FOUND AT: silent (below the ladder) on the merge path for the stage0 mirrors, the drop being declared but the step name overstating scope. RUNG NOW: still silent on the merge path; the step name now states its narrow population (gunbc.compiler_gate_workflow compiler_gate_drift_step), which removes the inflation but not the gap. ATTAINABLE CEILING: mechanically preventable (drift is a comparison against the authority's emission and cannot be made unwritable while the mirrors are committed bytes). NEXT-RUNG TRIGGER, stated as the capability: every committed stage0 mirror is refused on drift by a required step on the revision that lands, reading ONE roster in which the compile_stage0 emitted population derives its membership into gunbc.generated_artifact (one roster, two producers). Re-enabling --required-lane build alone does not retire this row: it restores the check and leaves the two rosters unjoined.",
],

evidence: [
DeclarationRef { module_path: "gunbc.generated_artifact", decl_name: "committed_generated_artifacts", field: WholeDeclaration },
DeclarationRef { module_path: "gunbc.compiler_gate_workflow", decl_name: "compiler_gate_drift_step", field: WholeDeclaration },
DeclarationRef { module_path: "gunbc.rung_drop.witness_floor_off_the_required_gate", decl_name: "witness_floor_off_the_required_gate", field: WholeDeclaration },
DeclarationRef { module_path: "gunbc.required_ci_phase_roster", decl_name: "RequiredCiPhase", field: WholeDeclaration },
],
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
module gunbc.recurring_failure_mode.required_floor_runs_at_its_memory_ceiling

import std.types { NonEmptyStr }
import std.decl_ref { DeclarationRef, WholeDeclaration }
import gunbc.recurring_failure_mode { RecurringFailureMode }

data required_floor_runs_at_its_memory_ceiling: RecurringFailureMode = RecurringFailureMode {
identity: "required_floor_runs_at_its_memory_ceiling" as NonEmptyStr,

receipts: [
"the required floor process already peaks AT its runner slot's memory.high, so it is memory-throttled on ordinary landings and has no headroom for its own growth (INVALID STATE: a required lane whose working set equals its slot's throttle line; HARM: the lane's wall time is partly throttling rather than work, so its duration measures nothing clean, and the next change that makes the floor heavier is throttled harder or OOM-killed rather than merely slower, taking a required lane down)",

"RECEIPT, re-derivable from the instrument rather than copied: the floor job's own [floor-cgroup] beats in merge-queue run 35716368507 (the #12029 landing, slot actions-runner@srv4-03). That slot carried memory.max=27917287424 and memory.high=26843545600. The floor process's slot peak was 26847006720, at the high line, with high=1109 throttle events, max=0 and oom_kill=0 on that run, and the job took about 40 minutes wall. Found in passing while pricing where the stage0 regen phases could run (gunbc.recurring_failure_mode.generated_population_split_across_two_unjoined_rosters); recorded separately because it is a standing property of main, not a consequence of that work.",

"WHY IT IS A WALL WITH NO MARGIN RATHER THAN A SLOW LANE. The classes of change this repository makes daily -- enrolling claims, widening a prepared population -- grow the floor's working set, and at the throttle line growth does not cost minutes, it costs the lane. And because an unknown share of the wall time is throttling, a speedup or slowdown of the floor cannot currently be attributed to work: the lane's duration is an instrument reading contaminated by the ceiling it runs against.",

"CONSEQUENCE FOR ANY STEP ADDED TO THE SAME JOB: another memory-heavy pass fits only as a SEPARATE process after the floor process exits (sequential peaks), never inside the floor's invocation (additive). That is a note, not a wall: claim_executor --required-ci with no --required-lane selects every phase in one process (required_ci_phase_selected with no lane answers true), so the additive form remains writable. With the floor at its ceiling, an invocation selecting all phases in the floor job is an OOM of a working required lane, and that is a consequence of the missing margin, not a defect in the CLI: running every phase in one process is an ordinary invocation, dangerous only because the floor has no headroom. Walling off the CLI would leave the no-margin condition ready to kill the next thing that grows; restoring headroom makes the additive form merely slow. The boundary this row names is the margin.",

"RUNG FOUND AT: silent -- no gate or diagnostic refuses a floor at its slot ceiling; the throttling is visible only in [floor-cgroup] beats nobody adjudicates. ATTAINABLE CEILING: mechanically preventable -- a required step can refuse a floor whose measured slot peak comes within a declared margin of memory.high, from the beats the run already writes. NEXT-RUNG TRIGGER, stated as the capability: the required run adjudicates the floor's measured slot peak against a declared headroom budget and refuses a landing that consumes it. UNRANKED: this row records the finding; no lane is dispatched for it.",
],

evidence: [
DeclarationRef { module_path: "gunbc.compiler_gate_workflow", decl_name: "compiler_gate_floor_run_step", field: WholeDeclaration },
],
}
22 changes: 18 additions & 4 deletions dag/gunbc/witness/compiler_gate_workflow.dag
Original file line number Diff line number Diff line change
Expand Up @@ -373,9 +373,23 @@ fn compiler_gate_floor_run_step() -> Step {
}
}

// THE STEP NAME STATES ITS POPULATION, AND THE POPULATION IS NARROWER THAN "GENERATED ARTIFACTS".
// generated_artifact_gate main reads gunbc.generated_artifact committed_generated_artifacts only. The
// ~157 stage0 mirrors (src/v1/stage0/src/std_measure.rs and the rest of the compile_stage0 emitted
// population) are adjudicated by claim_executor's GeneratedArtifact phase, which routes to
// --required-lane build, and no job has invoked that lane since #11742. The earlier name,
// "Generated artifacts match their authorities", stayed GREEN over that gap and read as a statement
// about stage0; std_measure.rs drifted on main under it (gunbc#12067). Declared drop:
// gunbc.rung_drop.witness_floor_off_the_required_gate. Row:
// gunbc.recurring_failure_mode.generated_population_split_across_two_unjoined_rosters.
//
// WHERE THE STAGE0 PHASES MAY RUN, measured rather than assumed: this job's floor process already
// peaks AT its slot's memory.high (run 35716368507: peak 26847006720 against high 26843545600,
// high events 1109), and the generated-artifact pass peaks near 12.9 GiB. They fit only as a
// SEPARATE process in a later step, after the floor exits -- never inside the floor's invocation.
fn compiler_gate_drift_step() -> Step {
RunStep {
name: Present { value: "Generated artifacts match their authorities" },
name: Present { value: "Registry-rostered generated artifacts (gunbc.generated_artifact) match their authorities; the stage0 mirrors are NOT checked here" },
id: Present { value: compiler_gate_drift_step_id },
run: concat(gunbc_ci_generated_artifact_verify_invoke(), "\n"),
shell: none,
Expand Down Expand Up @@ -433,7 +447,7 @@ fn compiler_gate_repair_declaration_step() -> Step {

fn compiler_gate_regen_step() -> Step {
RunStep {
name: Present { value: "Regenerate every committed generated artifact, then verify the tree it wrote" },
name: Present { value: "Regenerate every registry-rostered generated artifact (not the stage0 mirrors), then verify the tree it wrote" },
id: Present { value: compiler_gate_repair_regen_step_id },
run: concat(gunbc_ci_generated_artifact_regen_verified_invoke(), "\n"),
shell: none,
Expand Down Expand Up @@ -531,7 +545,7 @@ fn compiler_gate_floor_bound_steps() -> List<CompilerGateBoundStep> {
CompilerGateBoundStep {
step: compiler_gate_drift_step(),
role: capability_neutral,
step_name: "Generated artifacts match their authorities",
step_name: "Registry-rostered generated artifacts (gunbc.generated_artifact) match their authorities; the stage0 mirrors are NOT checked here",
},
CompilerGateBoundStep {
step: compiler_gate_repair_declaration_step(),
Expand All @@ -541,7 +555,7 @@ fn compiler_gate_floor_bound_steps() -> List<CompilerGateBoundStep> {
CompilerGateBoundStep {
step: compiler_gate_regen_step(),
role: capability_neutral,
step_name: "Regenerate every committed generated artifact, then verify the tree it wrote",
step_name: "Regenerate every registry-rostered generated artifact (not the stage0 mirrors), then verify the tree it wrote",
},
CompilerGateBoundStep {
step: compiler_gate_candidate_step(),
Expand Down