diff --git a/ROADMAP.md b/ROADMAP.md index 5be6c6168c6..cec7aa33f0a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -100,6 +100,8 @@ A flaky or green-but-broken floor means no gate protects anything — so CI is u **The <1min lever is resolve-phase incrementality, NOT cross-run resolve-cache** — CI is cold-dominated (the exe-hash re-colds on every code change), so §2 P3 resolve-cache is the *warm* ~18% lever and was measured net-negative as the cold-CI path. Incrementality (don't resolve what didn't change) is upstream of caching (memoize what repeats); the <1min target rides the former. **This re-frames the §2 "core ask" — operator review.** +→ [resolved-graph representation minimization](docs/plans/representation-minimization.md) — operator-funded root-first leanness after the #5867 InternTable fix (whole-tree resolve 14.2 to 5.5 GiB): emitter-determinism gate, stream/evict (Lever C — converges with resolve-incrementality above), variant-Node minimal representation, func_env.sigs single-authority. Gated on the root-vs-doomed-seed discriminator (v1 is going away). + **What's on `.dag` today (the gap map — detail in the charter §2/§4):** - [x] **execution = a dependency-graph walk** — `claim_executor` interprets `ci_floor_plan.dag`; one fold, batches from dependency edges (the realest layer) diff --git a/docs/plans/representation-minimization.md b/docs/plans/representation-minimization.md new file mode 100644 index 00000000000..c1378f3db50 --- /dev/null +++ b/docs/plans/representation-minimization.md @@ -0,0 +1,27 @@ +# Resolved-graph representation minimization (root-first, v1-survivable) + +Status: operator-funded 2026-06-27. Steer: minimal representation, attack root causes as much as we can, v1 is going away. This is the resolved-graph REPRESENTATION-minimization follow-on to the acute floor-OOM fix (#5867 shared InternTable: whole-tree resolve 14.2 GiB to 5.5 GiB, 61 percent reclaimed). It repoints the orphaned tidy-wren-707 resolved-graph-minimization lane. Distinct from `docs/plans/resource-aware-scheduler.md`, which is the ALLOCATION lane; this lane is REPRESENTATION minimization at the root. + +## The discriminator: root authority vs doomed seed + +v1 is going away, so every item is gated on one test: does the fix live in the `.dag` authority (and therefore survive into the v2 self-host), or is it a hand-edit to the doomed v1 Rust seed (and therefore die with v1)? Root items live in this plan. v1-only polish is explicitly de-scoped to bank-if-cheap and is NOT a pillar. + +## Current footprint + +Whole-tree resolve is about 5.5 GiB for about 7.5 MB of source: roughly 730x source-to-resolved. That is the width-1 all-modules-live worst case; the sharded CI floor holds only a subset per shard and sits under cap post-#5867. The footprint is now LINEAR with no quadratic OOM. Target: 20 to 50x. The path is structural (the levers below), not another big dedup: the func_env.sigs second-quadratic is real but only hundreds-of-MiB because its heavy provenance payloads are already Rc-shared (established by code read of `declared_to_resolved` and `merge_scope_from_imports`, not a probe). + +## Sequencing (root-first, v1-survivable) + +1. Emitter determinism is THE GATE for Lever B, and it is its own lane: PR #5879 (stern-fox-585) is the single authority for the gate's status and mechanism — this plan does not re-own it, it sequences on it. The dependency this plan asserts: while emit is non-reproducible, the standing workaround hand-edits the v1 Rust seed after each emit, which FORCES every representation change into the doomed v1 Rust instead of the `.dag` authority. So until #5879 lands the bit-identical fixed point (the §7 self-host last mile, see `regen-verify-gate.md`), the Node-representation change (Lever B) cannot be done at the root in `.dag` and regenerated. Gate for B; status lives in #5879, not here. +2. Lever C, stream/evict, is the biggest mover for the all-at-once figure and has NO dependency on the emitter gate. The 5.5 GiB is the whole-tree-all-modules-live worst case; resolve in dependency batches and evict a module env once its dependents are satisfied. Shares its root with the prune-resolve / affected_set selector lane (do not hold the whole corpus resolved at once), so author them as ONE batching-and-eviction model, not two. Better-ROI entry point; resolution-strategy fix at the root. +3. Lever B, variant-Node minimal representation, comes AFTER the gate. The Node is an 18-field superset struct; a leaf variable uses 3 fields and pays for about 18 (roughly 76 percent of the struct is unused per leaf, times about 627k nodes). Model a minimal per-kind sum type in `std/` (the `.dag` authority) and emit it, so the change regenerates rather than being hand-edited into the v1 seed (hence gated on item 1). The operator's minimal representation at the root; deep, broad blast radius. +4. func_env.sigs single-authority can land anytime. Each importing module re-materializes its transitive-closure signatures (merge_scope_from_imports plus rc_map_merge plus re-resolve), allocating fresh outer ResolvedFuncSig structs, a fresh map backbone, and duplicated raw function-name keys per module; the heavy provenance payloads are already Rc-shared. Fix: resolve each function signature ONCE and share the Rc, or look up imports via parent_index instead of merge-and-re-resolve, and intern the name keys. Same single-authority move as #5867. Moderate payoff, about hundreds of MiB. +5. Cheap hygiene is bank-if-cheap, NOT a pillar, because it lives in the v1 Rust seed and dies with v1: shared empty-Vec singleton (landed, about 43.7 MiB) and interning Node.name. Correction of record: Node.ident is NOT dead. An earlier reflection-count claim of 0-percent-used was refuted by execution; ident is a live ident-keyed fast-path on module and import nodes. Do not delete it. + +## Provenance + +Byte-attribution receipts: quiet-gull-17 (node-struct walk), fierce-bear-302 (env-map and payload walks, both probes reverted clean), silent-moth-532 (the intern-table root and the #5867 fix, three receipts: unique InternTable Rc 1397 to 1, byte-identical diagnostic fingerprint, 14.2 to 5.5 GiB). The func_env.sigs verdict was established by code read of the conversion sites, not by a probe. + +## Dissolution trigger (DESIGN §6) + +§7 v2 self-host fixed point: when the resolved-graph representation is minimal-by-construction in the `.dag` authority and the v1 Rust seed is retired, items 1 through 4 are landed-or-moot and item 5 dies with the seed; this lane dissolves into the v2 representation. diff --git a/dsl/gunbc/plan_registry_batch_g.dag b/dsl/gunbc/plan_registry_batch_g.dag index d9dc96e6b01..2bedaa3c43c 100644 --- a/dsl/gunbc/plan_registry_batch_g.dag +++ b/dsl/gunbc/plan_registry_batch_g.dag @@ -7,6 +7,7 @@ import gunbc.plans.resolver_type_name_collision_wall { resolver_type_name_collis import gunbc.plans.realization_measurement_loop { realization_measurement_loop_plan } import gunbc.plans.host_effect_orchestration { host_effect_orchestration_plan } import gunbc.plans.ci_floor_fractal_gantt { ci_floor_fractal_gantt_plan } +import gunbc.plans.representation_minimization { representation_minimization_plan } data batch_g_plans: List = [ ci_humming_plan, @@ -14,5 +15,6 @@ data batch_g_plans: List = [ resolver_type_name_collision_wall_plan, realization_measurement_loop_plan, host_effect_orchestration_plan, - ci_floor_fractal_gantt_plan + ci_floor_fractal_gantt_plan, + representation_minimization_plan ] diff --git a/dsl/gunbc/plans/representation_minimization.dag b/dsl/gunbc/plans/representation_minimization.dag new file mode 100644 index 00000000000..06d8160dcfe --- /dev/null +++ b/dsl/gunbc/plans/representation_minimization.dag @@ -0,0 +1,34 @@ +module gunbc.plans.representation_minimization + +import gunbc.plan { Plan, DissolutionTrigger, HasTrigger } +import std.markdown { MarkdownBlock } +import gunbc.plans.md_helpers { h2, p, li, ol } + +fn representation_minimization_body() -> List { + [ + p(text: "Status: operator-funded 2026-06-27. Steer: minimal representation, attack root causes as much as we can, v1 is going away. This is the resolved-graph REPRESENTATION-minimization follow-on to the acute floor-OOM fix (#5867 shared InternTable: whole-tree resolve 14.2 GiB to 5.5 GiB, 61 percent reclaimed). It repoints the orphaned tidy-wren-707 resolved-graph-minimization lane. Distinct from `docs/plans/resource-aware-scheduler.md`, which is the ALLOCATION lane; this lane is REPRESENTATION minimization at the root."), + h2(text: "The discriminator: root authority vs doomed seed"), + p(text: "v1 is going away, so every item is gated on one test: does the fix live in the `.dag` authority (and therefore survive into the v2 self-host), or is it a hand-edit to the doomed v1 Rust seed (and therefore die with v1)? Root items live in this plan. v1-only polish is explicitly de-scoped to bank-if-cheap and is NOT a pillar."), + h2(text: "Current footprint"), + p(text: "Whole-tree resolve is about 5.5 GiB for about 7.5 MB of source: roughly 730x source-to-resolved. That is the width-1 all-modules-live worst case; the sharded CI floor holds only a subset per shard and sits under cap post-#5867. The footprint is now LINEAR with no quadratic OOM. Target: 20 to 50x. The path is structural (the levers below), not another big dedup: the func_env.sigs second-quadratic is real but only hundreds-of-MiB because its heavy provenance payloads are already Rc-shared (established by code read of `declared_to_resolved` and `merge_scope_from_imports`, not a probe)."), + h2(text: "Sequencing (root-first, v1-survivable)"), + ol(items: [ + li(text: "Emitter determinism is THE GATE for Lever B, and it is its own lane: PR #5879 (stern-fox-585) is the single authority for the gate's status and mechanism — this plan does not re-own it, it sequences on it. The dependency this plan asserts: while emit is non-reproducible, the standing workaround hand-edits the v1 Rust seed after each emit, which FORCES every representation change into the doomed v1 Rust instead of the `.dag` authority. So until #5879 lands the bit-identical fixed point (the §7 self-host last mile, see `regen-verify-gate.md`), the Node-representation change (Lever B) cannot be done at the root in `.dag` and regenerated. Gate for B; status lives in #5879, not here."), + li(text: "Lever C, stream/evict, is the biggest mover for the all-at-once figure and has NO dependency on the emitter gate. The 5.5 GiB is the whole-tree-all-modules-live worst case; resolve in dependency batches and evict a module env once its dependents are satisfied. Shares its root with the prune-resolve / affected_set selector lane (do not hold the whole corpus resolved at once), so author them as ONE batching-and-eviction model, not two. Better-ROI entry point; resolution-strategy fix at the root."), + li(text: "Lever B, variant-Node minimal representation, comes AFTER the gate. The Node is an 18-field superset struct; a leaf variable uses 3 fields and pays for about 18 (roughly 76 percent of the struct is unused per leaf, times about 627k nodes). Model a minimal per-kind sum type in `std/` (the `.dag` authority) and emit it, so the change regenerates rather than being hand-edited into the v1 seed (hence gated on item 1). The operator's minimal representation at the root; deep, broad blast radius."), + li(text: "func_env.sigs single-authority can land anytime. Each importing module re-materializes its transitive-closure signatures (merge_scope_from_imports plus rc_map_merge plus re-resolve), allocating fresh outer ResolvedFuncSig structs, a fresh map backbone, and duplicated raw function-name keys per module; the heavy provenance payloads are already Rc-shared. Fix: resolve each function signature ONCE and share the Rc, or look up imports via parent_index instead of merge-and-re-resolve, and intern the name keys. Same single-authority move as #5867. Moderate payoff, about hundreds of MiB."), + li(text: "Cheap hygiene is bank-if-cheap, NOT a pillar, because it lives in the v1 Rust seed and dies with v1: shared empty-Vec singleton (landed, about 43.7 MiB) and interning Node.name. Correction of record: Node.ident is NOT dead. An earlier reflection-count claim of 0-percent-used was refuted by execution; ident is a live ident-keyed fast-path on module and import nodes. Do not delete it."), + ]), + h2(text: "Provenance"), + p(text: "Byte-attribution receipts: quiet-gull-17 (node-struct walk), fierce-bear-302 (env-map and payload walks, both probes reverted clean), silent-moth-532 (the intern-table root and the #5867 fix, three receipts: unique InternTable Rc 1397 to 1, byte-identical diagnostic fingerprint, 14.2 to 5.5 GiB). The func_env.sigs verdict was established by code read of the conversion sites, not by a probe."), + ] +} + +data representation_minimization_plan: Plan = Plan { + slug: "representation-minimization", + title: "Resolved-graph representation minimization (root-first, v1-survivable)", + body: representation_minimization_body(), + dissolution: HasTrigger { + text: "§7 v2 self-host fixed point: when the resolved-graph representation is minimal-by-construction in the `.dag` authority and the v1 Rust seed is retired, items 1 through 4 are landed-or-moot and item 5 dies with the seed; this lane dissolves into the v2 representation." + } +} diff --git a/dsl/gunbc/roadmap_authority.dag b/dsl/gunbc/roadmap_authority.dag index f41ff0826ac..11c64613927 100644 --- a/dsl/gunbc/roadmap_authority.dag +++ b/dsl/gunbc/roadmap_authority.dag @@ -158,6 +158,7 @@ fn section_1() -> RoadmapSection { edges: [], ), section_prose(content: "**The <1min lever is resolve-phase incrementality, NOT cross-run resolve-cache** — CI is cold-dominated (the exe-hash re-colds on every code change), so §2 P3 resolve-cache is the *warm* ~18% lever and was measured net-negative as the cold-CI path. Incrementality (don't resolve what didn't change) is upstream of caching (memoize what repeats); the <1min target rides the former. **This re-frames the §2 \"core ask\" — operator review.**"), + section_prose(content: "→ [resolved-graph representation minimization](docs/plans/representation-minimization.md) — operator-funded root-first leanness after the #5867 InternTable fix (whole-tree resolve 14.2 to 5.5 GiB): emitter-determinism gate, stream/evict (Lever C — converges with resolve-incrementality above), variant-Node minimal representation, func_env.sigs single-authority. Gated on the root-vs-doomed-seed discriminator (v1 is going away)."), section_group( label: "What's on `.dag` today (the gap map — detail in the charter §2/§4)", nodes: [