Repository navigation
Raise fleet per-slot row to 16GiB/15GiB from a re-measured floor peak - #8388
Conversation
…peak The 14GiB/13GiB row was derived from floor-job cgroup peaks (2026-08-05, p50 10.32GiB). The floor's peak moved: the claim-scope cut measures 15.6GB (14.53GiB), which does not fit a 14GiB MemoryMax. Same subject, changed input, so the derivation is re-run rather than repudiated. Observed on srv1 2026-08-17: all five slots carried the declared 14/13, and srv1-04/srv1-05 both peaked at EXACTLY 13.00GiB against 829 and 44803 memory.high events with zero oom_kills - a peak equal to the armed throttle line is the censored lower bound this module already taught, not demand. This is NOT readback reconciliation. The refusal stands unamended; the row moved because a measurement moved. That the target coincides with srv2-03's drifted readback is recorded as a coincidence, not a precedent. - per-slot 14/13 -> 16/15; widths srv3/srv4 7 -> 6 (fleet 24 -> 22) - new wall conjunct: MemoryHigh >= measured floor peak, so a row the floor cannot fit in is unwritable rather than discovered on a converged host - srv1 throttle observation + floor-peak carriers, each carrying the armed line beside the peak - srv2-03's drift is dissolved by the ruling; its two drift assertions and witnesses are deleted with a receipt, and a synthetic drifted fixture keeps the readback-refusal machinery witnessed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Finding 1: the readback-refusal note still said placement sizes against 'the declared 13GiB' after this PR moved MemoryHigh to 15GiB - a stale citation in the same file that edited the number. Now names the symbol. Finding 2 went further than the suggested both-bounds refusal, because the fleet already discriminates the readings by measurement. Five srv2 slots armed at 16106127360 report ZERO memory.high events, so their peaks are uncensored: max 15509315584 bytes. The binary reading (16750372454) exceeds the armed line and would have pinned them, so it is refuted by observation rather than declined by preference. The hand-typed 15600000000 is retired as the gate input and kept only as corroboration - nothing gates on it. Adds gunbc_floor_measured_peak_is_uncensored to the wall so a throttle-pinned (censored) peak cannot be used as the ceiling's input at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reviewer found the downstream sweep I had flagged as unfinished. All confirmed against current code, plus two they did not list. Witness pins moved 7 -> 6: - runner_slot_allocation_witness_test: host_memory_admitted_width + slots_per_host - runner_slot_provision_witness_test: srv4 runner_count AND srv3's seven-slot row (not in the review; same defect) - runner_host_deploy_witness_test: six names, srv4-07 now asserted ABSENT - host_allocation_conservation_test: width pins, derived-width probe, and the boundary control - srv3 refuses at 7 now, not 8 - runner_lifecycle_witness_test: committed width AND the disk-preflight requested_width fixtures (not in the review; same defect) Prose: - runner_host_deploy srv4_runner_count_disk_cap_note: 'memory admits 7' and the width-7 disk charge - srv2_03_converge_floor_fit_note: its open question is now ANSWERED, not merely stale - the 14/13 envelope was measured against the floor and failed. Rewritten as a superseded-with-receipt note rather than deleted, and the same stale claim removed from the refusal note that quoted it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
All seven findings in review 53001 verified against the code and fixed in Fixed as reported: width pins in Two the review did not list, same defect class:
One thing that needed more than a number change. On finding 7 ( Note the earlier review 52998 finding about the hand-typed peak literal is also resolved in |
incident_2026_08_06_roster_note asserted 'live fleet_desired_width_total is 24 after srv4 commits 7'. That function is DERIVED, so it already tracks gunbc_runner_slots_per_host and now reads 22 - only the prose was stale. Also names a numeric coincidence explicitly: the 2026-08-06 incident desired width was 22 (5+5+7+5, srv4 still row-pinned) and the current live total is also 22 (5+5+6+6). Same number, different quantity, and nothing should read the incident figure as the present one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
My blanket requested_width 7->6 sed hit the field but not its sibling _count fields, leaving requested_width: 6 against counts of 7. Since host_runner_capacity_converged requires every count to equal requested_width, the witness evaluated false and would have redded. Fixed by removing the coupling rather than re-syncing the number. The fixture's subject is convergence-shape PARITY, not the fleet's committed width, so binding it to the live width is what let a width ruling break it. All five fields now derive from one let-bound w, so they cannot desync again, and w is deliberately 3 - not any host's width - so nothing re-couples them. Also adds the missing RED control. host_runner_capacity_converged had only a positive witness, so an implementation returning true unconditionally passed. The control asserts a receipt one short on online_count does not converge, and again on materialized_count, so the conjunction is exercised at more than one field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed in the commit above. Review 53006 is correct and the defect was mine — a blanket I took neither of the two offered options, and it's worth saying why. The review noted the test's subject is convergence-shape parity rather than slot count — that's the actual root, so I removed the coupling instead of re-syncing the number:
I also added the RED control that was missing: Verified the rest of that sed's blast radius: the four disk-preflight fixtures each changed |
…iew 53008 prose
CI ROOT CAUSE: the ci job did not fail on this change - it failed at its
fail-fast gate because heal_generated_artifacts had already redded. That job
regenerated .github/workflows/{ci,fleet-converge,falsifier}.yml with
CARGO_BUILD_JOBS 3 -> 4 and could not push them (its credential cannot write
.github/workflows/**), so they are author-commit-required.
The bump is CORRECT and derived, not incidental: ci_cargo_build_jobs_count
reads ci_runner_target_ram_speed_budget, which under SlotCarved is the slot
RAM ceiling - so raising the per-slot cap 14 -> 16GiB raises the compile-jobs
derivation. I should have regenerated when the cap moved.
Review 53008 prose sweep, all five verified:
- host_disk_admitted_width reason string and the unknown-host note in
runner_slot_allocation (modeled output, not commentary)
- widths_are_pinned_on_both_axes_note and Subjects A-D in the witness file,
which described a 14/13 row and an srv2-03 drift the tests no longer assert
- derived_width_probe_note's 5/5/7/7 literals
Subject D was rewritten rather than renumbered: its question (is the floor
compliant with the modeled budget?) is now ANSWERED and the answer is no.
Subjects B and C now record that srv2-03 stopped discriminating and that the
refusal predicate's witness is deliberately synthetic.
Also caught one the review did not list: 'srv4 commits 7 like srv3 at
allocation' in the srv4 admission note, unmarked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Pushed CI root cause — not a defect in the changeThe The bump is derived and correct, not incidental: Review 53008 — all five verified and fixedThe two in Subject D was rewritten rather than renumbered. Its question — is the floor compliant with the modeled budget? — is no longer pending; it's answered, and the answer is no. An uncensored 15,509,315,584-byte peak sits above the superseded 14 GiB max, and srv1 pinned at exactly 13.00 GiB with 829 and 44,803 One the review did not list: Standing note on where the risk actually is: the 16/15 derivation has held up across four reviews. Every defect found so far has been in the mechanical propagation of the width change (7 → 6), which has a much wider blast radius than the cap change. That's the part worth the closest look. |
…2 slice cap (#8393) Both found by review 53011 against #8388 after it merged. Both execute on main today and both go red, which would block the fleet job's live_deploy step (it needs ci.result == success) and so block srv1 converging to 16/15. 1. incident_fleet_desired_width_at_outage pinned srv4 alone and read gunbc_runner_slots_per_host LIVE for srv1-srv3, so a 2026-08-06 historical total tracked present-day allocation. srv3 moving 7 -> 6 made it return 21 against a corpus and witness that both say 22. Now every host is pinned by date (5/5/7/5), which is what DESIGN section 5 requires of a historical measurement compared to a literal - half-pinned was the defect. 2. witness_srv2_slice_cap_drifts_after_per_host_recarve expected Drifted. srv2's pool budget is now 5 x 16GiB = 85899345920, EXACTLY the untouched 2026-07-03 fixture, so the verdict is legitimately Converged again. Renamed and flipped, with a RED control asserting it still drifts against the superseded 70GiB cap so the comparison keeps a failing direction. The srv2 note records why flip two was legitimate and why it is NOT the dissolve-on the previous revision predicted: no readback replaced the fixture and nothing was applied to srv2. The declaration moved back to meet a slice carved at 80GiB all along - which is also why srv2 never needed converging on this axis. Neither the fixture nor the declaration was edited toward the other. srv1's slice witness is unaffected: its fixture is infinity, which drifts against any finite budget. Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…is the cut Three conflicting paths, three different questions, resolved separately. ci.yml and falsifier.yml are modify/delete: this branch deleted them, main edited them. Taking main's side would silently un-delete the exact thing the branch exists to remove, and it would do it with no conflict marker for review to catch. The deletion stands, and this is not a fresh ruling — their .dag authorities were deleted with them, so nothing on this branch emits either file and restoring the bytes would leave a generated artifact with no producer. Main's edit to them is dispositioned rather than dropped unread: it is #8388, raising the per-slot row to 16GiB/15GiB from a re-measured floor peak. That change arrives on this branch through its own authority in the .dag side of this merge, which is where the fleet declaration actually lives; the workflow YAML was only a projection of it. So nothing of #8388 is lost by keeping the two files deleted — the projection they carried has no consumer here, because witnesses.yml declares no per-slot memory of its own. fleet-converge.yml is the only one of the three that is an ordinary merge, and it is a generated projection, so it was not resolved by picking a side: the driver refused (correctly — both sides moved since the base, so neither side's bytes are the projection of the merged authorities), and it was regenerated. main_wet, then regen_stage0, then a rebuild and regen_stage0 --verify on a binary that postdates the change it emits: regen_divergence_count=0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 53053 requested changes: the previous commit replaced two fixture literals with expressions derived from the same input the plan under test consumes, which weakens the oracle toward measure() == measure(). That is the right objection and the repair was the wrong one. The actual defect was never the literal — it was that the fixture read the LIVE fleet row as its input, so #8388's width 7 -> 6 silently changed the expected value and redded main. A literal beside a moving input is the change detector; a literal beside a planted input is the controlled fixture DESIGN §5 names as the legitimate first arm. So both witnesses now construct their own five-slot RunnerHostDeploy and assert exact counts against it: 5 desired - 2 observed = 3 added, and 5 desired - 1 observed = 4 upserts. Every number is authored, none is computed from the other, and the three quantities are pairwise distinct so a coincidence cannot satisfy the assertion. The live fleet width keeps its own coverage, unchanged and independent: witness_srv3_deploy_row_names_six_slots and witness_srv4_runner_count_six_materialization_target still read the real rows and still pin 6. What the fixtures no longer do is restate that width as a side effect of testing the reconcile arithmetic. Field literals follow the model's own deploy rows (bare strings into the NonEmptyStr fields), not a refinement cast.
* Derive the srv3 provision witness counts from the declared width The 7 -> 6 width change in #8388 left two literals behind in runner_slot_provision_witness_test.dag, and they redded main's floor: 2 of 9978 discovery witnesses failed, both in this file. witness_membership_reconcile_adds_delta_slots: 2 observed against a 6-slot desired set adds 4, not 5. witness_provision_plan_extracts_only_upserts: 1 observed against a 6-slot desired set upserts 5, not 6. Both now read the count off desired_runner_slot_members and the fixture's own observed list rather than restating it, so the next width change cannot desync them. The first test already carried the derived form as a second conjunct beside the stale literal; that literal is what broke, and the derived conjunct is what survives. This is not a loosening: the fixture plants an exact observed population and the claim is the reconcile arithmetic over it, not the width. Width itself stays pinned by witness_srv3_deploy_row_names_six_slots and witness_srv4_runner_count_six_materialization_target in the same file. * Plant the width too, so the reconcile counts are exact literals again Review 53053 requested changes: the previous commit replaced two fixture literals with expressions derived from the same input the plan under test consumes, which weakens the oracle toward measure() == measure(). That is the right objection and the repair was the wrong one. The actual defect was never the literal — it was that the fixture read the LIVE fleet row as its input, so #8388's width 7 -> 6 silently changed the expected value and redded main. A literal beside a moving input is the change detector; a literal beside a planted input is the controlled fixture DESIGN §5 names as the legitimate first arm. So both witnesses now construct their own five-slot RunnerHostDeploy and assert exact counts against it: 5 desired - 2 observed = 3 added, and 5 desired - 1 observed = 4 upserts. Every number is authored, none is computed from the other, and the three quantities are pairwise distinct so a coincidence cannot satisfy the assertion. The live fleet width keeps its own coverage, unchanged and independent: witness_srv3_deploy_row_names_six_slots and witness_srv4_runner_count_six_materialization_target still read the real rows and still pin 6. What the fixtures no longer do is restate that width as a side effect of testing the reconcile arithmetic. Field literals follow the model's own deploy rows (bare strings into the NonEmptyStr fields), not a refinement cast. --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com>
…iB -> 15GiB This is the value correction, isolated so it can be signed separately from the lens that found it. The seed constant is a MIRROR of gunbc.runner_slot_allocation gunbc_runner_slot_desired field memory_high. That row moved to 16106127360 (15.0 GiB) in 936f451 / #8388 on 2026-08-17 -- "Raise fleet per-slot row to 16GiB/15GiB from a re-measured floor peak" -- and the mirror did not follow, so the seed carried the superseded 13958643712 (13.0 GiB) for eleven days. This is convergence, not a fleet-axis decision: a committed mirror may only move toward its authority, and the judgement was made by #8388. DIRECTION, stated plainly. The constant has one use site, uncapped_host_budget_from_mem_available, which caps an uncapped host's MemAvailable sample at the declared throttle line. The governor was therefore capping 2 GiB BELOW the declared line; this RAISES arming to what placement and ci_budget_tree already size against. Source-level reading of the use site and its doc comment -- not executed, and not a claim about observed fleet behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iB -> 15GiB This is the value correction, isolated so it can be signed separately from the lens that found it. The seed constant is a MIRROR of gunbc.runner_slot_allocation gunbc_runner_slot_desired field memory_high. That row moved to 16106127360 (15.0 GiB) in 936f451 / #8388 on 2026-08-17 -- "Raise fleet per-slot row to 16GiB/15GiB from a re-measured floor peak" -- and the mirror did not follow, so the seed carried the superseded 13958643712 (13.0 GiB) for eleven days. This is convergence, not a fleet-axis decision: a committed mirror may only move toward its authority, and the judgement was made by #8388. DIRECTION, stated plainly. The constant has one use site, uncapped_host_budget_from_mem_available, which caps an uncapped host's MemAvailable sample at the declared throttle line. The governor was therefore capping 2 GiB BELOW the declared line; this RAISES arming to what placement and ci_budget_tree already size against. Source-level reading of the use site and its doc comment -- not executed, and not a claim about observed fleet behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the 11-day fleet arming drift it caught (13GiB -> 15GiB) (#8638) * Seed-mirror constant lens: join the five hand-Rust budget constants to their authority rows Two seed_mirror_notes each asked for "one lens reading the seed's constants against their cited rows" and neither was built. They also named DIFFERENT overlapping subsets, so the mechanism had two partial specifications and no implementation. This is the union. The lens never writes an expected value: each assertion renders its literal from the imported authority row, so editing the lens cannot satisfy it and editing the authority moves the expectation. An uncited or unparseable constant REFUSES rather than skipping -- "nothing to check, so green" is the empty-observation narrow, and of the five constants in scope the one that carried no citation is the one that drifted. Also canonicalizes the in-scope seed literals to plain decimal (3 * 1024 * 1024, 4_000, 12_884_901_888) and adds the missing authority citations. Values unchanged in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Converge DECLARED_RUNNER_SLOT_MEMORY_HIGH_BYTES to its authority: 13GiB -> 15GiB This is the value correction, isolated so it can be signed separately from the lens that found it. The seed constant is a MIRROR of gunbc.runner_slot_allocation gunbc_runner_slot_desired field memory_high. That row moved to 16106127360 (15.0 GiB) in 936f451 / #8388 on 2026-08-17 -- "Raise fleet per-slot row to 16GiB/15GiB from a re-measured floor peak" -- and the mirror did not follow, so the seed carried the superseded 13958643712 (13.0 GiB) for eleven days. This is convergence, not a fleet-axis decision: a committed mirror may only move toward its authority, and the judgement was made by #8388. DIRECTION, stated plainly. The constant has one use site, uncapped_host_budget_from_mem_available, which caps an uncapped host's MemAvailable sample at the declared throttle line. The governor was therefore capping 2 GiB BELOW the declared line; this RAISES arming to what placement and ci_budget_tree already size against. Source-level reading of the use site and its doc comment -- not executed, and not a claim about observed fleet behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Derive the host-budget witness expectation from its authority; retire both notes' future tense Two inert-by-construction surfaces sat beside the drift, which is why eleven days passed without either catching it. The witness: test.claim.host_budget_source built BudgetSourceMemAvailableCappedAtDeclaredHigh with byte_size(13958643712) -- the mirror value under test, retyped into the expectation -- so it greened whichever way the mirror went. A measurement copied from the tree under test is not an oracle. It now reads gunbc_runner_slot_desired().memory_high. The notes: both seed_mirror_notes described the covering lens in the future tense while nothing executed. A dissolution trigger naming a mechanism that does not run reads as guarded to every later author. Both now describe what executes, and state what remains undissolved -- the lens proves the mirror AGREES, not that it is unnecessary, so the class sits at mechanically preventable rather than structurally impossible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fold the seed-mirror roster instead of re-listing rows; close the reach gap the enumeration hid neat-bee-14 refuted this PR's own claim by reading the diff: it said an uncited in-scope constant REFUSES rather than skips, but "in scope" was five hand-enumerated assertions plus five hardcoded symbol names. A new seed mirror constant was therefore neither checked NOR refused -- invisible, exactly like the drift this module exists to catch. Shipping that sentence would have reproduced the defect inside its own fix. The rows are now one folded roster. Forward: every row must find its declaration and authority symbol in the seed. Backward: marker occurrences per seed file must equal roster rows homed in that file. Rows are distinct and each must be present, so agreeing counts mean no marked constant exists that the roster does not carry -- set equality, not a count standing in for one. The backward count folds this module's own roster, so automating it does not collapse to measure() == measure(). Standardizes the marker across all five declarations, and states the residual plainly: a constant carrying NO marker is still invisible to both arms. This raises the bar from "remembered to edit a list in another file" to "marked the declaration where it was written", and names the next-rung trigger as a Rust-declaration-level census. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use the compiling list idiom: list_length/split pipe and fold, not filter(f:) The substrate refused filter(xs:, f:) -- it declares 'predicate', not 'f'. Mirrors the idiom from e0599_emitter_decision_census_witness_test rather than inventing one, and replaces the filter+length count with a fold. Note filesystem_path_grant absolute_path_segments calls filter(xs:, f:) and is the shape that misled me; that is a separate finding, not touched here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Pin four calls to string_contains: my import shadowed the bare name they used CI caught a regression I caused. Importing gunbc.runner_slot_allocation to derive the declared_high expectation pulled v2.std.algebra into this module's closure, and its contains(xs, item, eq) captured the bare name four unrelated lines were using for the string contains(haystack, needle). v1-seed fn names are not module-scoped, so an import added for one declaration re-resolved a name used by different declarations in the same file, with no local edit to those lines. The floor fold aborted during preparation, so the entire roster failed to run over a two-line edit elsewhere in the file. string_contains is unambiguous and is what this PR's own lens already uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Record the pending sixth row in the module, not only in session chat The obligation to enroll #8635's DECLARED_WHOLE_CORPUS_COMPILE_MEASURED_DEMAND_BYTES lived only in messages between two sessions, either of which can archive. Anything that must outlive a session belongs in a carrier. Also records WHY that constant is unmarked rather than merely unrostered: the backward arm reds on a marked constant with no row, so a marker landing before its row would red main for the interval between merges. Marker and row land together, in whichever PR is second. Adding the marker alone is the one repair that must not happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Correct the rung: the lens is discovered and DECLINED, so the class is mitigatable Measured, not inferred. Floor run 32337765205 at 4806220 reports claims=9782 declined_live=782; run 32333231183 at the same base reports claims=9782 declined_live=778. Adding this module moved declined_live by exactly +4 and claims by 0, so discovery finds it and the hermetic fold declines it. The declaration is correct: the lens reads .rs text, which is live host state, so ReadsLiveTree is true and SubstrateInputsOnly would be a lie -- and the falsifier that used to catch lying rows died in the floor cut. So the honest rung is MITIGATABLE. The lens discriminates by execution (three arms, 8573608) but nothing runs it automatically, which is exactly the tier DESIGN section 6 names: machinery exists, nothing gates on it. Both seed_mirror_notes are corrected to say so. Larger than this module: 778 live-tree witnesses were already declined at this PR's base. The whole class of seed-.rs-versus-.dag-row check is unexecutable by the current fold, so the covering mechanism both notes asked for cannot be enforced by CI as architected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
What
gunbc_runner_slot_desiredmoves from 14GiB/13GiB to 16GiB/15GiB, widths on srv3/srv4 from 7 to 6 (fleet 24 -> 22).Why, and why this is not readback reconciliation
The 14/13 row was derived from floor-job cgroup peaks (ruling 2026-08-05: 186 peaks, p50 10.32GiB). The floor's peak moved - the claim-scope cut measures 15.6GB (14.53GiB), which does not fit a 14GiB MemoryMax and sits 1.53GiB above a 13GiB MemoryHigh. Same subject, changed input, so the derivation is re-run.
gunbc_runner_slot_allocation_refuses_effective_readback_reconciliationstands unamended. That the new target equals srv2-03's drifted readback is recorded in-carrier as a coincidence, explicitly not a precedent.Observed evidence (srv1, 2026-08-17)
All five srv1 slots carried the declared 14/13. Two were pinned:
A peak equal to the armed throttle line is the censored lower bound this module already taught for the four 15.0GiB pins in its own calibration. Zero oom_kills does not mean the envelope fit - reclaim and swap absorbed it (10GiB swap in use).
Corroboration across the fleet: srv2/srv3/srv4 (all at 16/15) show clean unthrottled peaks of 14.41 / 14.46 GiB, consistent with the decimal reading of 15.6 GB.
Construction
New wall conjunct
MemoryHigh >= gunbc_floor_measured_peak_observationmakes a per-slot row the floor cannot fit in unwritable, rather than discoverable at 3am on a converged host. Two independently derived quantities, per this wall's own oracle standard.Consequences handled
gunbc_synthetic_drifted_cap_observationreplaces it, deliberately synthetic so no future convergence can green it.Stated honestly
fleet-convergeapply would read main's 14/13 as desired and revert the fleet.Verification
No local compile check: the session binary reports 1594 errors on clean main (too stale to parse
//annotations). Relies on CI.