From f2f1f9eaf65581ff74e817cff83029e23ec24fdc Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 14:53:41 +0000 Subject: [PATCH 1/7] =?UTF-8?q?docs/plans:=20compute-envelope-model=20?= =?UTF-8?q?=E2=80=94=20one=20authority=20for=20the=20CI=20fleet's=20resour?= =?UTF-8?q?ce=20dimensions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan doc resolving the §1 ROADMAP "CI on compute fabric" pointer. Models the bimodal crash-or-idle pathology as one root (N hand-tuned resource dimensions, no single ResourceEnvelope authority) and the §3 fix: derive every knob — spawn_width, fan-out, TasksMax, jobserver, MemoryMax — from one measured envelope, with the public(shape)/ctrl(realization) split. Co-owned warm-lark-306 + quick-ant-298 (§1 lead); CC bright-stag-194 (ROADMAP + test profile). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/plans/compute-envelope-model.md | 130 +++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/plans/compute-envelope-model.md diff --git a/docs/plans/compute-envelope-model.md b/docs/plans/compute-envelope-model.md new file mode 100644 index 00000000000..18d6a36438e --- /dev/null +++ b/docs/plans/compute-envelope-model.md @@ -0,0 +1,130 @@ +# Compute envelope — one authority for the CI fleet's resource dimensions + +> Plan doc. Resolves the §1 ROADMAP pointer **"CI on compute fabric"**. Co-owned: **warm-lark-306** +> (authoring + operator context) and **quick-ant-298** (§1 CI-floor lead; owns the spawn-width slice). +> CC **bright-stag-194** (ROADMAP owner + test-profile owner for the fan-out cap). DESIGN refs: §1 +> (time = cost/safety), §2 (Realization — the parallelization arm), §3 (single authority + the +> measured-peripheral split), §5 (fail-closed; *derive*, don't hand-set). +> +> A few live numbers below are marked **[confirm]** — quick-ant has the freshest values from the §1 +> investigation; the *model* is the point, not those scalars. + +## 1. The symptom: a fleet with no operating point + +Two 128-core hosts (srv1, srv2) that are **either oversubscribed/crashing or sitting at ~1% doing +nothing — never the sane middle.** That bimodal swing *is* the diagnosis. A system that swings between +starvation and thrashing, with no stable middle, is a system with **no modeled operating point**: +nothing answers "how much work fits this host," so the fleet falls to one extreme or the other. + +Live evidence (srv1, 2026-06-21): + +- **Under side (~1%):** 128 cores, loadavg **1.27** (~1% util), **9 / 125 GB** mem used, 1 + `Runner.Worker` executing but **0 rustc / 0 cargo**. The box is nearly empty during CI. +- **Over side (crash):** a single debug build's thread fan-out (`CARGO_BUILD_JOBS=15` × rustc + codegen-units 256 × LLVM × sccache) bursts **one** runner cgroup past `TasksMax=4096` → `clone()` + rejected by the pids controller → **EAGAIN** false build failure — a spike on an otherwise-idle host. + +Neither extreme is a capacity problem. The host has enormous headroom on both axes; the failures are +**coordination** failures. + +## 2. The root: N hand-tuned dimensions, no single authority (§3) + +Every resource/parallelism knob is set **independently, by a different hand, blind to the others** — +the textbook §3 violation (one fact, many homes). There is no `ResourceEnvelope` the knobs derive from, +so they cannot be mutually consistent: + +| Dimension | Set today by | Blind to | +| --- | --- | --- | +| floor **spawn width** | `min(shard_count, cores)` (`std/realization_width.dag`) + memory term (#5444, merging) | the true core count (envelope says 64c — **a lie**, it's 128c); pids cap | +| **host packing** (PRs/host) | implicit — ~one PR's CI runs at a time across 50 runner units | the idle 120+ cores | +| per-build **fan-out** | `CARGO_BUILD_JOBS` × codegen-units (debug 256) × LLVM | the pids cap it then bursts | +| cgroup **pids cap** | `TasksMax=4096` hand-set drop-in (ctrl) | the fan-out that legitimately needs >4096 | +| **jobserver tokens** | `~120` magic constant (ctrl) | cores, the pids cap | +| cgroup **`MemoryMax`** | per-host hand edit — **drifted** (srv1 ≠ srv2) [confirm] | the other host | +| watchdog / isolation / runner count | hand-set systemd | the envelope | + +The under-side and the over-side are the **same defect** seen twice: width bounded *below* the envelope +(idle) and fan-out bursting *above* a cap that the envelope never informed (crash). + +## 3. The move: one ResourceEnvelope, every knob derived (§3 single authority) + +Collapse the N hand-tuned dimensions into **one authority** — the host `ResourceEnvelope` (cores, +memory, pids capacity; already a type in `dsl/product/compute_fabric.dag`) — and make **every knob a +function of it**. Then the dimensions stop being N things to juggle: they are one measured fact with +derivations, and the fleet gets **exactly one coherent operating point**. It *cannot* be both starved +and thrashing, because both the width floor and the fan-out ceiling come from the same envelope. + +| Knob | Derivation from the envelope | Tier (§4) | Owner | Status | +| --- | --- | --- | --- | --- | +| measured host facts | **the envelope itself** — cores, mem, pids capacity, *measured* | ctrl realization | ctrl | model is **wrong** (64c/256GB; real 128c/128GB) — **fix first** | +| spawn width | `min(shard_demand, cores, mem_budget ÷ per-shard-peak, pids_budget ÷ per-shard-pids)` — generalize `bounded_host_spawn_width` (today `min(shard_count, cores)`) with the mem term (#5444) + the pids term, and raise `shard_demand` so width isn't bottlenecked below the envelope | public | quick-ant (slice) | §2 below | +| per-build fan-out | codegen-units / `CARGO_BUILD_JOBS` capped so one build's pids ≤ `pids_cap ÷ peak_concurrent_builds` | public (test profile) | bright-stag (#5456 area) | §3 below | +| pids cap (`TasksMax`) | `≥ peak_per_build_pids × safety`, bounded by `kernel_threads_max ÷ runner_count` | ctrl realization | ctrl | tourniquet applied (4096→16384, 2026-06-21); should become **derived** | +| jobserver tokens | a function of cores (the cooperative compile limit), consistent with the pids cap | ctrl realization | ctrl | derive, don't magic-constant | +| `MemoryMax` | one value from the envelope, identical across same-spec hosts | ctrl realization | ctrl | de-drift | + +**The invariant that forecloses both failures:** width floor and fan-out ceiling are derived from the +**same** envelope ⇒ `peak_concurrent_pids = spawn_width × per_build_pids ≤ pids_cap` holds *by +construction* (§5), not by hope. You can't tune width up into a crash, because the same envelope that +raised width also sized the cap and the fan-out. + +## 4. The PUBLIC / CTRL split (§3 measured-peripheral — the doc's spine) + +Per §3, the agnostic **shape** is central; the **measured realization** and the **dispatch** are +peripheral. Applied here: + +- **PUBLIC (central, topology):** the envelope *shape*, the width *derivation*, the codegen-units cap — + all live in `dsl/product/compute_fabric.dag` / `std/realization_width.dag` / the `[profile.test]` + block. These are host-agnostic functions: "given an envelope, here is the width / the fan-out." +- **CTRL (peripheral, realization):** the *measured host facts* (128c/128GB), `TasksMax`, `MemoryMax`, + runner count, the systemd-cgroup knobs, the unit generation. These are srv1/srv2 *instances* of the + shape. **gunbc owns the shapes; ctrl owns the instances and *generates* the units** from the public + derivation (the units are emitted, never hand-typed — that's what kills the drift). + +So the doc maps both tiers, but routes implementation accordingly: width/fan-out → public PRs; +host-fact grounding + unit generation → ctrl. + +## 5. Sequencing (operator wants this ASAP) + +1. **Ground the envelope in measured truth** — kill the 64c/256GB lie (real: 128c/128GB, both hosts). + *Everything downstream inherits this; you cannot derive width from a lying envelope.* (ctrl + the + `compute_fabric` host facts.) +2. **Spawn width up** — derive width from the grounded envelope (cores ∧ mem ∧ pids), raising it toward + the host's real capacity instead of the low bound that leaves 120 cores idle. Builds on **#5444** + (memory term) + **#5421/#5375** (per-node demand, even-width removal, memory-aware — *already + landed*); this is a **refinement**, not a re-derivation. **Owned by quick-ant** (single slice — see + §6; do **not** fork). Current effective width = **[quick-ant confirm]**. +3. **Cap the per-build fan-out** — codegen-units, so one build can't burst the pids cap. Composes into + **one** `[profile.test]` block with bright-stag's **#5456** opt-level work (not a fork). +4. **(2) and (3) derive from the same envelope** ⇒ the coherent operating point. Then generate the ctrl + units (`TasksMax`/`MemoryMax`/jobserver) from the envelope too, retiring the hand-set drop-ins. + +The top lever is **(2)**: it converts the idle 120 cores into real utilization — the biggest single +CI-slow win. (1) is its precondition. + +## 6. Ownership + the no-fork rule + +- **spawn width** is **quick-ant's lane** — a single slice on top of #5444/#5421. `warm-crane-135` + (an earlier candidate owner) is **archived** (closed 2026-06-20); there is no parallel item, and one + must not be opened — a forked width fix would be the exact §3 violation this doc exists to fix. + Note: `adhoc-240256ec-32b` is **done** (it = merged #5421); the remaining "derive width up / + host-packing" work needs its own correctly-scoped item — **[quick-ant to identify/open]**. +- **fan-out cap** → bright-stag (test-profile owner), composed with #5456. +- **host-fact grounding + unit generation** → ctrl. +- **this doc + the §1 ROADMAP bullets** → bright-stag applies (author→owner pattern); quick-ant + co-signs the §1 substance. + +## 7. Landed / in-flight context (so this builds on, not re-derives) + +- **#5421** (merged) — per-node resource demand + stop the executor's even-width division (CI 24m→~10m). +- **#5375** (merged) — memory-aware floor scheduling. +- **#5419** (merged) — pinned floor width (the bound (2) raises). +- **#5444** (merging) — width value from measured-RAM-budget ÷ measured-per-shard-peak (the **memory + term** of the envelope derivation). +- **#5431** (merged) — the measured-peak source the memory term reads. +- **#5456** (merged) — `[profile.test] opt-level=3` for `v1-compiler` (the fan-out cap composes here). +- **TasksMax 4096→16384** applied on srv1+srv2 (2026-06-21) — the tourniquet; should become *derived*. + +This is the §2 **parallelization-by-realization** arm (Schedule/Placement/Width of +`std/realization.dag`) made coherent with its measured substrate — the structural twin of the §2 +caching arm, both derived from one authority. From 2233246dbb943bef79795d4e6742ff01c7b10864 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 14:56:04 +0000 Subject: [PATCH 2/7] compute-envelope-model: correct current width = pinned 4 (quick-ant verified) Memory-aware width model was removed as unwired (#5419); current floor width is a pinned constant 4 (ci_floor_plan.dag:292), ~3% of 128c. #5444 re-adds the memory term (not yet on main). The lever lifts both the pin and the shard_count cap from the envelope. Fresh spawn-width slice to be reopen-scoped by quick-ant. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/plans/compute-envelope-model.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/plans/compute-envelope-model.md b/docs/plans/compute-envelope-model.md index 18d6a36438e..c65b9e44f9e 100644 --- a/docs/plans/compute-envelope-model.md +++ b/docs/plans/compute-envelope-model.md @@ -35,7 +35,7 @@ so they cannot be mutually consistent: | Dimension | Set today by | Blind to | | --- | --- | --- | -| floor **spawn width** | `min(shard_count, cores)` (`std/realization_width.dag`) + memory term (#5444, merging) | the true core count (envelope says 64c — **a lie**, it's 128c); pids cap | +| floor **spawn width** | a **pinned constant 4** (`gunbc_ci_floor_spawn_width`, `ci_floor_plan.dag:292`) — the memory-aware derivation was *removed* as unwired (#5419); #5444 (not yet on main) re-adds the memory term | the true core count (envelope says 64c — **a lie**, it's 128c); the ~124 idle cores; the pids cap | | **host packing** (PRs/host) | implicit — ~one PR's CI runs at a time across 50 runner units | the idle 120+ cores | | per-build **fan-out** | `CARGO_BUILD_JOBS` × codegen-units (debug 256) × LLVM | the pids cap it then bursts | | cgroup **pids cap** | `TasksMax=4096` hand-set drop-in (ctrl) | the fan-out that legitimately needs >4096 | @@ -57,7 +57,7 @@ and thrashing, because both the width floor and the fan-out ceiling come from th | Knob | Derivation from the envelope | Tier (§4) | Owner | Status | | --- | --- | --- | --- | --- | | measured host facts | **the envelope itself** — cores, mem, pids capacity, *measured* | ctrl realization | ctrl | model is **wrong** (64c/256GB; real 128c/128GB) — **fix first** | -| spawn width | `min(shard_demand, cores, mem_budget ÷ per-shard-peak, pids_budget ÷ per-shard-pids)` — generalize `bounded_host_spawn_width` (today `min(shard_count, cores)`) with the mem term (#5444) + the pids term, and raise `shard_demand` so width isn't bottlenecked below the envelope | public | quick-ant (slice) | §2 below | +| spawn width | `min(shard_demand, cores, mem_budget ÷ per-shard-peak, pids_budget ÷ per-shard-pids)` — replace the **pinned 4** with a derivation that lifts **both** the pin **and** the `shard_demand` cap from the envelope (cores ∧ mem ∧ pids), building on #5444's mem term (+#5421's layer-at-once, done) so the corpus actually uses the 128c/115GB-free headroom | public | quick-ant (slice) | §5 below | | per-build fan-out | codegen-units / `CARGO_BUILD_JOBS` capped so one build's pids ≤ `pids_cap ÷ peak_concurrent_builds` | public (test profile) | bright-stag (#5456 area) | §3 below | | pids cap (`TasksMax`) | `≥ peak_per_build_pids × safety`, bounded by `kernel_threads_max ÷ runner_count` | ctrl realization | ctrl | tourniquet applied (4096→16384, 2026-06-21); should become **derived** | | jobserver tokens | a function of cores (the cooperative compile limit), consistent with the pids cap | ctrl realization | ctrl | derive, don't magic-constant | @@ -90,10 +90,12 @@ host-fact grounding + unit generation → ctrl. *Everything downstream inherits this; you cannot derive width from a lying envelope.* (ctrl + the `compute_fabric` host facts.) 2. **Spawn width up** — derive width from the grounded envelope (cores ∧ mem ∧ pids), raising it toward - the host's real capacity instead of the low bound that leaves 120 cores idle. Builds on **#5444** - (memory term) + **#5421/#5375** (per-node demand, even-width removal, memory-aware — *already - landed*); this is a **refinement**, not a re-derivation. **Owned by quick-ant** (single slice — see - §6; do **not** fork). Current effective width = **[quick-ant confirm]**. + the host's real capacity instead of the low bound that leaves ~124 cores idle. **Current width is a + pinned constant 4** (`ci_floor_plan.dag:292`; the memory-aware model was removed as unwired in #5419) + — on 128c that is ~3% of cores from width alone. The lever lifts **both** the pin **and** the + `shard_count` cap on the derivation. Builds on **#5444** (memory term, re-adds it; not yet on main) + + **#5421/#5375** (per-node demand, even-width removal — *already landed*); this is a **refinement**, + not a re-derivation. **Owned by quick-ant** (single fresh slice — see §6; do **not** fork). 3. **Cap the per-build fan-out** — codegen-units, so one build can't burst the pids cap. Composes into **one** `[profile.test]` block with bright-stag's **#5456** opt-level work (not a fork). 4. **(2) and (3) derive from the same envelope** ⇒ the coherent operating point. Then generate the ctrl @@ -107,8 +109,10 @@ CI-slow win. (1) is its precondition. - **spawn width** is **quick-ant's lane** — a single slice on top of #5444/#5421. `warm-crane-135` (an earlier candidate owner) is **archived** (closed 2026-06-20); there is no parallel item, and one must not be opened — a forked width fix would be the exact §3 violation this doc exists to fix. - Note: `adhoc-240256ec-32b` is **done** (it = merged #5421); the remaining "derive width up / - host-packing" work needs its own correctly-scoped item — **[quick-ant to identify/open]**. + Note: `adhoc-240256ec-32b` is **done** (it = merged #5421) — *not* the open slice; no existing + open item covers "derive width up (unpin + lift the `shard_count` cap) + host-packing across PRs", so + quick-ant **reopen-scopes a fresh slice** (confirmed), sequenced after envelope-grounding (#1) and + built on #5444 + #5421. - **fan-out cap** → bright-stag (test-profile owner), composed with #5456. - **host-fact grounding + unit generation** → ctrl. - **this doc + the §1 ROADMAP bullets** → bright-stag applies (author→owner pattern); quick-ant From d6144ec3bdabb58939768c2119acad96a5fc8bb4 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 14:57:53 +0000 Subject: [PATCH 3/7] =?UTF-8?q?ROADMAP=20=C2=A71:=20point=20the=20CI-on-fa?= =?UTF-8?q?bric=20chore-line=20at=20the=20compute-envelope=20plan=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes #5462 self-contained (doc + its own pointer, atomic, no orphan window). Different line from the §1 nightly-reframe edits — no collision. Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index f5b7c4bb7a3..808656c9584 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -69,7 +69,7 @@ here: it's the substrate CI runs on; selling it as an infra piece is downstream. - [ ] **floor runs reliably & affordably** — memory-aware scheduling (spawn_width is memory-blind → deterministic OOM as the corpus grows; `ResourceEnvelope.memory` modeled but unwired) and kill build flakes (sccache corruption ⇒ false-green: exit-0 with no artifact) - [ ] **tree-scoped builtin availability (registry partition)** (fail-closed) — the seed `builtin_function_registry` (76 names, `src/v1/04_method.dag`, a marked bridge scaffold) is **global, not scoped to the compiled tree**, so v1-seed intrinsics leak into the dsl-substrate compile and *resolve without a real `.dag` def* (a §5 fail-open, surfaced by `utf8_decode_bytes` in `secret_manager.dag` — the gate is green despite no genuine definition). Fix by construction: the substrate may use only builtins with real `.dag` defs; seed-only names admitted only when entry-root = the v1 seed — this advances the registry's *own* sanctioned dissolve-on ("deleted when builtins are actual `.dag` definitions"). Load-bearing seed + a likely red wave → **measure-first** (leaked-name count: how many of the 76 the substrate relies on), expose-then-triage, escalate before the enforce-flip. *Owned by §1 (quick-ant-298); instance fix (real `utf8_decode_bytes` std fn in `std/encoding` **+** removal of the registry bridge entry & its seed mirror) = **#5452** (verified sound: symbol now resolves via the real `.dag` def, execution is the fail-closed `v1_rt::utf8_decode_bytes` intercept; fleet-gated). Class fix (tree-scoped partition) still open.* - [ ] repo model (internal repo) on compute fabric -- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory- & pids-aware) · per-build fan-out (`CARGO_BUILD_JOBS`×codegen-units×LLVM) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). +- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory- & pids-aware) · per-build fan-out (`CARGO_BUILD_JOBS`×codegen-units×LLVM) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). - [ ] *(downstream / expansion)* compute fabric as a sellable infra piece ## 2. Minimal work — caching by realization (fail-closed) From 62901b1b8205ae84bca2cba0d150370e7ed61b67 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 15:00:39 +0000 Subject: [PATCH 4/7] compute-envelope-model: split RUN-phase width from BUILD-phase pids (quick-ant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §3 invariant conflated two phases. spawn_width is the discovery-corpus RUN-phase shard width (witnesses against the prebuilt binary — cores∧mem-bound, pids-light), so it does NOT multiply per_build_pids; width-up is pids-safe on its own. The pids crash is the BUILD phase: concurrent_builds × per_build_pids ≤ pids_cap, coupling host-packing × fan-out, not spawn_width. Two invariants, not one product. Also: #5375 memory-aware was superseded by #5419's pin (#5444 re-adds the term). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/plans/compute-envelope-model.md | 65 +++++++++++++++++++--------- 1 file changed, 44 insertions(+), 21 deletions(-) diff --git a/docs/plans/compute-envelope-model.md b/docs/plans/compute-envelope-model.md index c65b9e44f9e..c4431ac1ba1 100644 --- a/docs/plans/compute-envelope-model.md +++ b/docs/plans/compute-envelope-model.md @@ -35,9 +35,9 @@ so they cannot be mutually consistent: | Dimension | Set today by | Blind to | | --- | --- | --- | -| floor **spawn width** | a **pinned constant 4** (`gunbc_ci_floor_spawn_width`, `ci_floor_plan.dag:292`) — the memory-aware derivation was *removed* as unwired (#5419); #5444 (not yet on main) re-adds the memory term | the true core count (envelope says 64c — **a lie**, it's 128c); the ~124 idle cores; the pids cap | -| **host packing** (PRs/host) | implicit — ~one PR's CI runs at a time across 50 runner units | the idle 120+ cores | -| per-build **fan-out** | `CARGO_BUILD_JOBS` × codegen-units (debug 256) × LLVM | the pids cap it then bursts | +| floor **spawn width** (RUN) | a **pinned constant 4** (`gunbc_ci_floor_spawn_width`, `ci_floor_plan.dag:292`) — the memory-aware derivation was *removed* as unwired (#5419); #5444 (not yet on main) re-adds the memory term | the true core count (envelope says 64c — **a lie**, it's 128c); the ~124 idle cores; the free memory | +| **host packing** (BUILD) | implicit — ~one PR's CI runs at a time across 50 runner units | the idle 120+ cores; the pids it jointly bursts with fan-out | +| per-build **fan-out** (BUILD) | `CARGO_BUILD_JOBS` × codegen-units (debug 256) × LLVM | the pids cap it bursts (× concurrent builds) | | cgroup **pids cap** | `TasksMax=4096` hand-set drop-in (ctrl) | the fan-out that legitimately needs >4096 | | **jobserver tokens** | `~120` magic constant (ctrl) | cores, the pids cap | | cgroup **`MemoryMax`** | per-host hand edit — **drifted** (srv1 ≠ srv2) [confirm] | the other host | @@ -57,16 +57,29 @@ and thrashing, because both the width floor and the fan-out ceiling come from th | Knob | Derivation from the envelope | Tier (§4) | Owner | Status | | --- | --- | --- | --- | --- | | measured host facts | **the envelope itself** — cores, mem, pids capacity, *measured* | ctrl realization | ctrl | model is **wrong** (64c/256GB; real 128c/128GB) — **fix first** | -| spawn width | `min(shard_demand, cores, mem_budget ÷ per-shard-peak, pids_budget ÷ per-shard-pids)` — replace the **pinned 4** with a derivation that lifts **both** the pin **and** the `shard_demand` cap from the envelope (cores ∧ mem ∧ pids), building on #5444's mem term (+#5421's layer-at-once, done) so the corpus actually uses the 128c/115GB-free headroom | public | quick-ant (slice) | §5 below | -| per-build fan-out | codegen-units / `CARGO_BUILD_JOBS` capped so one build's pids ≤ `pids_cap ÷ peak_concurrent_builds` | public (test profile) | bright-stag (#5456 area) | §3 below | +| spawn width (**RUN** phase) | `min(shard_demand, cores, mem_budget ÷ per-shard-peak)` — replace the **pinned 4** with a derivation that lifts **both** the pin **and** the `shard_demand` cap from the envelope (cores ∧ mem), building on #5444's mem term (+#5421's layer-at-once, done) so the corpus actually uses the 128c/115GB-free headroom. Discovery shards run witnesses against the **prebuilt** release binary — light on pids, no rustc — so **no pids term**; width-up is pids-safe on its own | public | quick-ant (slice) | §5 below | +| host packing (**BUILD** phase concurrency) | `concurrent_builds` — how many PRs' builds co-run on a host; the factor that multiplies the build-phase pids fan-out (see invariant) | public/ctrl | quick-ant + ctrl | §5 below | +| per-build fan-out (**BUILD** phase) | codegen-units / `CARGO_BUILD_JOBS` capped so `concurrent_builds × per_build_pids ≤ pids_cap` | public (test profile) | bright-stag (#5456 area) | §5 below | | pids cap (`TasksMax`) | `≥ peak_per_build_pids × safety`, bounded by `kernel_threads_max ÷ runner_count` | ctrl realization | ctrl | tourniquet applied (4096→16384, 2026-06-21); should become **derived** | | jobserver tokens | a function of cores (the cooperative compile limit), consistent with the pids cap | ctrl realization | ctrl | derive, don't magic-constant | | `MemoryMax` | one value from the envelope, identical across same-spec hosts | ctrl realization | ctrl | de-drift | -**The invariant that forecloses both failures:** width floor and fan-out ceiling are derived from the -**same** envelope ⇒ `peak_concurrent_pids = spawn_width × per_build_pids ≤ pids_cap` holds *by -construction* (§5), not by hope. You can't tune width up into a crash, because the same envelope that -raised width also sized the cap and the fan-out. +**The invariant that forecloses both failures — but mind the two phases (they don't multiply each +other):** a PR's CI is a **BUILD** phase (rustc fans out, once) followed by a **RUN** phase (the +discovery corpus shards run witnesses against the *prebuilt* binary). They touch different resources: + +- **RUN phase — spawn width** is cores ∧ mem bound. Discovery shards barely touch pids (no rustc), so + width does **not** multiply the build fan-out. **Width-up is pids-safe on its own** — you can raise it + toward the envelope without crash risk. (Premise: shards consume the prebuilt binary — exactly what + #5450's Pop-B build-once enforces; a shard that still rebuilds is a *defect to fix*, not a reason to + pids-bound width.) +- **BUILD phase — the pids crash** is `concurrent_builds × per_build_pids ≤ pids_cap`. This couples + **host-packing** (how many PRs build at once) with **per-build fan-out** (codegen-units) — *not* + spawn_width. That is where by-construction crash-prevention lives. + +So the same-envelope coupling is **packing × fan-out** (build phase); width-up (run phase) is an +*independent* cores∧mem lever. You can't tune either into a crash, because each is bounded by the +envelope on its own axis — but they are **two invariants, not one product.** ## 4. The PUBLIC / CTRL split (§3 measured-peripheral — the doc's spine) @@ -89,17 +102,25 @@ host-fact grounding + unit generation → ctrl. 1. **Ground the envelope in measured truth** — kill the 64c/256GB lie (real: 128c/128GB, both hosts). *Everything downstream inherits this; you cannot derive width from a lying envelope.* (ctrl + the `compute_fabric` host facts.) -2. **Spawn width up** — derive width from the grounded envelope (cores ∧ mem ∧ pids), raising it toward - the host's real capacity instead of the low bound that leaves ~124 cores idle. **Current width is a - pinned constant 4** (`ci_floor_plan.dag:292`; the memory-aware model was removed as unwired in #5419) - — on 128c that is ~3% of cores from width alone. The lever lifts **both** the pin **and** the - `shard_count` cap on the derivation. Builds on **#5444** (memory term, re-adds it; not yet on main) + - **#5421/#5375** (per-node demand, even-width removal — *already landed*); this is a **refinement**, - not a re-derivation. **Owned by quick-ant** (single fresh slice — see §6; do **not** fork). -3. **Cap the per-build fan-out** — codegen-units, so one build can't burst the pids cap. Composes into - **one** `[profile.test]` block with bright-stag's **#5456** opt-level work (not a fork). -4. **(2) and (3) derive from the same envelope** ⇒ the coherent operating point. Then generate the ctrl - units (`TasksMax`/`MemoryMax`/jobserver) from the envelope too, retiring the hand-set drop-ins. +2. **Spawn width up (RUN phase)** — derive width from the grounded envelope on **cores ∧ mem** (no pids + term — discovery shards run the *prebuilt* binary), raising it toward the host's real capacity instead + of the low bound that leaves ~124 cores idle. **Current width is a pinned constant 4** + (`ci_floor_plan.dag:292`; the memory-aware model was removed as unwired in #5419) — on 128c that is + ~3% of cores from width alone. The lever lifts **both** the pin **and** the `shard_count` cap. **This + is pids-safe on its own** (run phase ≠ build phase — see the §3 invariant), so width can go up + independently of the crash-side discipline. Builds on **#5444** (memory term, re-adds it; not yet on + main) + **#5421** (even-width removal — *landed*); a **refinement**, not a re-derivation. **Owned by + quick-ant** (single fresh slice — see §6; do **not** fork). *(Premise: shards consume the prebuilt + binary — #5450 Pop-B build-once; a rebuilding shard is a defect to fix, not a reason to pids-bound + width.)* +3. **Cap the per-build fan-out + bound host-packing (BUILD phase)** — codegen-units so + `concurrent_builds × per_build_pids ≤ pids_cap`. This is the crash-side discipline: it couples + host-packing (#2's BUILD-phase concurrency) with fan-out, **not** spawn_width. Codegen-units composes + into **one** `[profile.test]` block with bright-stag's **#5456** opt-level work (not a fork). +4. **(2) and (3) each derive from the same envelope on their own axis** ⇒ the coherent operating point — + *two* invariants (run-phase cores∧mem; build-phase packing×fan-out≤pids), not one product. Then + generate the ctrl units (`TasksMax`/`MemoryMax`/jobserver) from the envelope too, retiring the + hand-set drop-ins. The top lever is **(2)**: it converts the idle 120 cores into real utilization — the biggest single CI-slow win. (1) is its precondition. @@ -121,7 +142,9 @@ CI-slow win. (1) is its precondition. ## 7. Landed / in-flight context (so this builds on, not re-derives) - **#5421** (merged) — per-node resource demand + stop the executor's even-width division (CI 24m→~10m). -- **#5375** (merged) — memory-aware floor scheduling. +- **#5375 → superseded by #5419** — #5375's memory-aware scheduling was removed as unwired complexity by + #5419's pin (#5375 < #5419, so #5419 reverted it); **#5444 re-adds** the memory term. Current state = + **pinned 4**, not memory-aware. - **#5419** (merged) — pinned floor width (the bound (2) raises). - **#5444** (merging) — width value from measured-RAM-budget ÷ measured-per-shard-peak (the **memory term** of the envelope derivation). From 1b4b0bb13ee0cc1c53dc9b6d068d75e1965a130b Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 15:13:53 +0000 Subject: [PATCH 5/7] =?UTF-8?q?ROADMAP=20=C2=A71:=20fix=20chore-line=20spa?= =?UTF-8?q?wn=5Fwidth=20conflation=20(RUN/BUILD=20split)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit main's compact bullet (#5459) said 'spawn_width (memory- & pids-aware)' — the same conflation quick-ant corrected in compute-envelope-model.md. spawn_width is RUN-phase (cores∧mem, prebuilt binary, pids-light); pids binds the BUILD phase (fan-out × concurrent-builds). Keeps ROADMAP consistent with the doc in one PR. Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 808656c9584..a04ceb2e4b7 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -69,7 +69,7 @@ here: it's the substrate CI runs on; selling it as an infra piece is downstream. - [ ] **floor runs reliably & affordably** — memory-aware scheduling (spawn_width is memory-blind → deterministic OOM as the corpus grows; `ResourceEnvelope.memory` modeled but unwired) and kill build flakes (sccache corruption ⇒ false-green: exit-0 with no artifact) - [ ] **tree-scoped builtin availability (registry partition)** (fail-closed) — the seed `builtin_function_registry` (76 names, `src/v1/04_method.dag`, a marked bridge scaffold) is **global, not scoped to the compiled tree**, so v1-seed intrinsics leak into the dsl-substrate compile and *resolve without a real `.dag` def* (a §5 fail-open, surfaced by `utf8_decode_bytes` in `secret_manager.dag` — the gate is green despite no genuine definition). Fix by construction: the substrate may use only builtins with real `.dag` defs; seed-only names admitted only when entry-root = the v1 seed — this advances the registry's *own* sanctioned dissolve-on ("deleted when builtins are actual `.dag` definitions"). Load-bearing seed + a likely red wave → **measure-first** (leaked-name count: how many of the 76 the substrate relies on), expose-then-triage, escalate before the enforce-flip. *Owned by §1 (quick-ant-298); instance fix (real `utf8_decode_bytes` std fn in `std/encoding` **+** removal of the registry bridge entry & its seed mirror) = **#5452** (verified sound: symbol now resolves via the real `.dag` def, execution is the fail-closed `v1_rt::utf8_decode_bytes` intercept; fleet-gated). Class fix (tree-scoped partition) still open.* - [ ] repo model (internal repo) on compute fabric -- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory- & pids-aware) · per-build fan-out (`CARGO_BUILD_JOBS`×codegen-units×LLVM) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). +- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory-aware; NOT pids-bound — pids binds the build phase, see next) · per-build fan-out × concurrent-builds ≤ pids cap (`CARGO_BUILD_JOBS`×codegen-units×LLVM) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). - [ ] *(downstream / expansion)* compute fabric as a sellable infra piece ## 2. Minimal work — caching by realization (fail-closed) From e539f97c44a54286ef15b11c9ab767142d55ca9c Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 15:14:49 +0000 Subject: [PATCH 6/7] =?UTF-8?q?ROADMAP=20=C2=A71:=20adopt=20quick-ant's=20?= =?UTF-8?q?precise=20spawn=5Fwidth=20de-conflation=20wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Explicit RUN/BUILD phase labels — spawn_width = cores∧mem (RUN, prebuilt binary, pids-light); pids burst belongs on per-build fan-out (BUILD, concurrent_builds × per-build-pids ≤ TasksMax). No "pids" on the spawn_width bullet at all (the conflation's verification check). Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index a04ceb2e4b7..76dee97f1a8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -69,7 +69,7 @@ here: it's the substrate CI runs on; selling it as an infra piece is downstream. - [ ] **floor runs reliably & affordably** — memory-aware scheduling (spawn_width is memory-blind → deterministic OOM as the corpus grows; `ResourceEnvelope.memory` modeled but unwired) and kill build flakes (sccache corruption ⇒ false-green: exit-0 with no artifact) - [ ] **tree-scoped builtin availability (registry partition)** (fail-closed) — the seed `builtin_function_registry` (76 names, `src/v1/04_method.dag`, a marked bridge scaffold) is **global, not scoped to the compiled tree**, so v1-seed intrinsics leak into the dsl-substrate compile and *resolve without a real `.dag` def* (a §5 fail-open, surfaced by `utf8_decode_bytes` in `secret_manager.dag` — the gate is green despite no genuine definition). Fix by construction: the substrate may use only builtins with real `.dag` defs; seed-only names admitted only when entry-root = the v1 seed — this advances the registry's *own* sanctioned dissolve-on ("deleted when builtins are actual `.dag` definitions"). Load-bearing seed + a likely red wave → **measure-first** (leaked-name count: how many of the 76 the substrate relies on), expose-then-triage, escalate before the enforce-flip. *Owned by §1 (quick-ant-298); instance fix (real `utf8_decode_bytes` std fn in `std/encoding` **+** removal of the registry bridge entry & its seed mirror) = **#5452** (verified sound: symbol now resolves via the real `.dag` def, execution is the fail-closed `v1_rt::utf8_decode_bytes` intercept; fleet-gated). Class fix (tree-scoped partition) still open.* - [ ] repo model (internal repo) on compute fabric -- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory-aware; NOT pids-bound — pids binds the build phase, see next) · per-build fan-out × concurrent-builds ≤ pids cap (`CARGO_BUILD_JOBS`×codegen-units×LLVM) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). +- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory-aware: cores ∧ mem — RUN phase; discovery shards run the prebuilt binary, pids-light) · per-build fan-out (`CARGO_BUILD_JOBS`×codegen-units×LLVM — BUILD phase, the pids burst: concurrent_builds × per-build-pids ≤ TasksMax) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). - [ ] *(downstream / expansion)* compute fabric as a sellable infra piece ## 2. Minimal work — caching by realization (fail-closed) From 27c153af7268cfb1842a59f6c51cdaf51a908581 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Sun, 21 Jun 2026 15:27:05 +0000 Subject: [PATCH 7/7] =?UTF-8?q?ROADMAP=20=C2=A71:=20shorten=20CI-on-fabric?= =?UTF-8?q?=20chore-line=20to=20a=20summary=20+=20doc=20pointer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per operator: ROADMAP lines stay scannable progress summaries; density goes in plan docs (its own stated rule — no restating detail). The 1175-char inline chore-list/diagnosis/formulas now live in compute-envelope-model.md; the line is a one-sentence summary + pointer. Density preserved in the doc, not lost. Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 76dee97f1a8..098809a08c2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -69,7 +69,7 @@ here: it's the substrate CI runs on; selling it as an infra piece is downstream. - [ ] **floor runs reliably & affordably** — memory-aware scheduling (spawn_width is memory-blind → deterministic OOM as the corpus grows; `ResourceEnvelope.memory` modeled but unwired) and kill build flakes (sccache corruption ⇒ false-green: exit-0 with no artifact) - [ ] **tree-scoped builtin availability (registry partition)** (fail-closed) — the seed `builtin_function_registry` (76 names, `src/v1/04_method.dag`, a marked bridge scaffold) is **global, not scoped to the compiled tree**, so v1-seed intrinsics leak into the dsl-substrate compile and *resolve without a real `.dag` def* (a §5 fail-open, surfaced by `utf8_decode_bytes` in `secret_manager.dag` — the gate is green despite no genuine definition). Fix by construction: the substrate may use only builtins with real `.dag` defs; seed-only names admitted only when entry-root = the v1 seed — this advances the registry's *own* sanctioned dissolve-on ("deleted when builtins are actual `.dag` definitions"). Load-bearing seed + a likely red wave → **measure-first** (leaked-name count: how many of the 76 the substrate relies on), expose-then-triage, escalate before the enforce-flip. *Owned by §1 (quick-ant-298); instance fix (real `utf8_decode_bytes` std fn in `std/encoding` **+** removal of the registry bridge entry & its seed mirror) = **#5452** (verified sound: symbol now resolves via the real `.dag` def, execution is the fail-closed `v1_rt::utf8_decode_bytes` intercept; fleet-gated). Class fix (tree-scoped partition) still open.* - [ ] repo model (internal repo) on compute fabric -- [ ] **CI on compute fabric** — centralize host management under the fabric: one host `ResourceEnvelope` is the single authority, every knob **derived** from it and **generated** (gunbc owns unit shapes, ctrl owns srv1/srv2 instances), not the hand-set systemd+shell that drifts and contradicts today (the EAGAIN clone-reject flake, spawn_width OOM, and `MemoryMax` drift are all this one un-modeled-envelope root). **Chore list to model — derive, don't hand-set:** measured host facts (the fleet model is *wrong*: says 64c/256GB, real is 128c/128GB) · cgroup pids cap (`TasksMax`) · jobserver token count · `spawn_width` (memory-aware: cores ∧ mem — RUN phase; discovery shards run the prebuilt binary, pids-light) · per-build fan-out (`CARGO_BUILD_JOBS`×codegen-units×LLVM — BUILD phase, the pids burst: concurrent_builds × per-build-pids ≤ TasksMax) · `MemoryMax` (drifted srv1≠srv2) · watchdog / isolation / runner topology. This is the **parallelization-by-realization** arm of `std/realization.dag` (the twin of §2 caching). Full dimension map + the derive-from-one-envelope plan: [compute-envelope-model](docs/plans/compute-envelope-model.md). +- [ ] **CI on compute fabric** — derive every host knob (spawn_width, fan-out, `TasksMax`, jobserver, `MemoryMax`) from one measured `ResourceEnvelope`; ends the crash-or-idle swing. → [compute-envelope-model](docs/plans/compute-envelope-model.md) - [ ] *(downstream / expansion)* compute fabric as a sellable infra piece ## 2. Minimal work — caching by realization (fail-closed)