diff --git a/ROADMAP.md b/ROADMAP.md index a2e2275e308..756ccb8bff3 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -121,7 +121,6 @@ A flaky or green-but-broken floor means no gate protects anything — so CI is u - [ ] **G2 runner deployment + cross-host placement** *(dispatched proud-tern-439: model + drift-gate)* — derive runners/host + registration from `fleet_intent`+envelope; **host-apply RESOLVED (operator-set, CI-humming plan authority): the gunbc-EMITTED converge shell** (`.github/fleet-converge.sh` = a §2 host-effect handler inhabiting the §6 carrier *by emission* — the emitted shell IS the executing consumer, escaping the §5 spec-without-execution trap; ctrl thin-runs it + parses a fail-closed receipt; §7 seed-shrink). In progress: emit (wise-eagle-664) + thin consume/run (fierce-carp) - [ ] **G3 cgroup caps** — `TasksMax`/`MemoryMax` host-set by hand, only *read* live; derive + reconcile via the **emitted converge shell + fail-closed receipt** (same carrier-inhabitation-by-emission as G2) so a hand-edit reds - [ ] **CI on compute fabric** — derive every host knob from one measured `ResourceEnvelope`; ends the crash-or-idle swing [plan](docs/plans/compute-envelope-model.md) -- [ ] **compute fabric connector migration** — relocate ~50 removed vocabulary symbols to their single-authority homes before #5889 can merge [plan](docs/plans/compute-fabric-connector-migration.md) **Adjacent gaps (smaller, outside the host band):** @@ -207,7 +206,7 @@ Anchor (do not flip-flop): `.dag` = truth; purely self-hosting (v2 emits its own Adjacent lane — algorithmic-cost rewrite engine (the §3 construction design; post-stability, natural once `.dag` is the self-hosted truth). → [plan](docs/plans/algebraic-rewrite-optimization.md) -**◆ Milestones (critical path):** front-end ✓ · emit-rust well-typed ✓ · de-fork Step 1 ✓ (#5473) · class-3 corpus-coherence + cargo-green seed ✓ (#5481) → **▸ NOW: emitted crate cargo-builds green** → std forks collapsed → real fixed point → **KEYSTONE: `regen --verify` in CI** → seed-honesty → **TERMINAL: `src/v1` deleted** *(§7 regen-fixpoint deferred, #5514; src/v1 NOT yet deletable)* +**◆ Milestones (critical path):** front-end ✓ · emit-rust well-typed ✓ · de-fork Step 1 ✓ (#5473) · class-3 corpus-coherence + cargo-green seed ✓ (#5481) → emitted crate cargo-builds green ✓ (#5777/#5873) → **▸ NOW: real fixed point** → `regen --verify` in CI ✓ (#5873) → seed-honesty → **TERMINAL: `src/v1` deleted** *(§7 regen-fixpoint deferred, #5514; src/v1 NOT yet deletable)* - [ ] rewrite suboptimal patterns to the cheaper equivalent (`O(n²)→O(n)` …) — published finite catalog, bulletproof where it fires - [ ] `Unknown` dissolved over time as an anemic atom (reuse the `Disposition` carrier) @@ -219,7 +218,7 @@ Adjacent lane — algorithmic-cost rewrite engine (the §3 construction design; - [x] **Root A — emit-seam grounding ✓ LANDED (#5734)**: String/`FreeMonoid`→host `Vec` via the #5428 `RustCorpusRepr` seam; drove HostNative cargo-green *(jolly-cat)* - [ ] **Root B — generic-inference keystone ✓ MERGED (#5552)** → def-unification (coproduct authority + aliases) → repoints (algebra/nat/integer/float/logic/effects/verification) → 🟡-marker dissolution *(bright-deer-111)* - [ ] v1-coupled `coercion`/`node` renames — deferred to v1-delete -- [ ] **emitted crate `cargo build`s green (Route-A last mile)** — execution-grounded, **re-measured after unmask** (#5714, deep-wolf, `--emit-fresh` full 108-emitted + 22-periphery closure): the earlier **38 E0308 was an E0433-masked LOWER BOUND** — true ceiling **647 E0308** (the §5-honesty call to re-measure-after-unmask was right, ~17x over the masked bound; *no* green was ever claimed against `38→0`). **Family A (24 map-literal `&str` keys) CLEARED → 0** (type-DERIVED key coercion, landed via the merged Lane-A re-baseline #5716/#5718 after the standalone #5714 was superseded/closed). **Pole = the Optional-on-String-carrier family: 638/647 (98.6%)** — and execution **REFUTED the first-cut 'pipe-first consumer-unwrap' root cause**: the dominant 638 are **signature-rendering** bugs (`expected String, found Option`), the Rust emitter **dropping the `Optional` wrapper on String text-carrier signature/field types** (the carrier path early-returns a bare `String` before the Optional-template renderer; `Int?` rendered `Option` correctly, only `String?` regressed). The function *body* already emits `Some/None` — only the *type* was unfaithful, so the model was right (`first` IS modeled `OptionalOf`, algebra.dag:295). Fix = **faithful Optional RENDERING** (`rust_carrier_optional_wrap`: wrap iff `return_cardinality == CardOptional`, cardinality-gated + a non-optional-stays-bare control) — #5716, **CONSTRUCTION not a coercion**. Trajectory by `--emit-fresh`: **647 → 9** (the 11-site renderer wrap) **→ 6** (3 fail-closed unwraps at the genuine `T?`-into-`T`-slot model-hole consumers — semver_compare_identifier, schedule_witness_entry_eq, runnable_eq — located `.expect`, never `unwrap_or_default`, the empty-None PANIC proven by catch_unwind against a fabricates-0 control). **E0308 POLE CLEARED: 647 → 0** (the final 6 — fn-item-as-value 1 @ semver.rs:226 · and-str 3 · Box/Rc DescentEvidence 2 — clear in deep-wolf's 6-tail PR; fn-item via a load-bearing record-collapse-fold consistency fix, gated). But **0 E0308 is a MILESTONE, not cargo-green** — execution **REFUTED the '0 E0308 = done' finish-line**: the lane measured E0308-ONLY (its brief said 'categorized E0308 count'), and the FULL cargo build shows **~32 OTHER pre-existing errors never counted**, constant across measures (orthogonal, not unmasked — 41=9+32, 38=6+32, 32=0+32). Re-census after the measure-tower pole CLEARED: **measure-Q LANDED (#5735)** — value-variant type-arg collapse, the E0560×12/E0614×7 `std_measure.rs` root — and **std_serialization C0 cleared (#5761)** (a fixed-point prereq). Full `--emit-fresh` cargo build now **5 TOTAL errors**; #5777 (open, merry-gull — E0599 `Clone`-bound + E0624 Optional-receiver) takes **5 → 2**, and the residual **2 are E0091×2** — one **Compose decision** away. So **cargo-green = 0 TOTAL cargo errors** (the real self-host build), no longer the measure pole. Model hole stays flagged: pipe-first is modeled `OptionalOf`, so a **model-forces-unwrap follow-up is owed UPSTREAM** (the Optional/`Value::Null` runway). **"done" = the emitted crate cargo-builds with 0 TOTAL errors (the E0308 milestone + the 32 cleared) + the PRs merged** — measure full-cargo, never a grep of one error family. *(jolly-cat #5705/#5716/#5718 + the 6-tail/32-baseline; supersedes the stale 38-lower-bound, the ~12,600 / ~150-latent estimate, the refuted pipe-first-unwrap root cause, AND the refuted '0 E0308 = cargo-green' finish-line)* +- [x] **emitted crate `cargo build`s green (Route-A last mile) ✓ LANDED (#5777/#5873)** — execution-grounded **re-census CONFIRMED 0 TOTAL errors** (cool-hawk-908: `regen_stage0 --emit-fresh` → `cargo build` debug+release, 2026-06-28). **REFUTED** the stale 'residual 2 = E0091×2' hypothesis: #5777 cleared E0599/E0624 **and** E0091 (`alias_unused_param_names` → `PhantomData` for unused type-alias params; `type_alias_phantom_param_test` witnesses). Trajectory: 647 E0308 → 0 (#5716/#5718/#5717) → ~32 other-family → 5 TOTAL → **0 TOTAL** (#5777). #5873: two-generation regen cutover + `regen --verify` CI gate. Compose emits `PhantomData<(Algebra, MachineConstraint)>`. Model hole stays flagged: pipe-first `OptionalOf` unwrap follow-up owed upstream (`Value::Null` runway). *(jolly-cat #5705/#5716/#5718 + cool-hawk-908 re-census)* - [ ] real fixed point: `content_hash` stage1==stage2 (dissolve placeholder hashes) - [x] wire `regen_stage0 --verify` lockstep gate into CI — enforces no hand-edits to **generated** stage0 files (`GENERATED_STAGE0_FILES`); `HAND_MAINTAINED_STAGE0_FILES` are copied through, excluded by design ← **keystone** - [ ] dissolve seed hand-patches (`patch_*` / `HAND_MAINTAINED_STAGE0_FILES`) — the `emit_rust` hand-sync caveat the gate reproduces (the required-facts plan is dissolved into the executable `RegenVerifyGate`) diff --git a/docs/plans/compute-fabric-connector-migration.md b/docs/plans/compute-fabric-connector-migration.md deleted file mode 100644 index 3def69d91be..00000000000 --- a/docs/plans/compute-fabric-connector-migration.md +++ /dev/null @@ -1,128 +0,0 @@ -# compute_fabric connector — vocabulary relocation tracking - -> Plan doc. Tracks every symbol removed from `product.compute_fabric` in the connector remodel ([PR #5889](https://github.com/gunb-ai/gunbc/pull/5889)). Framing: the connector is the thin matching law (need × opportunity → connection); the removed vocabulary becomes the connector's grounding once it relocates to its single-authority home. Disposition column: **RELOCATE** = has live consumers, needs a new home before #5889 can merge; **DELETE** = fabric-internal with no external consumers. Status checkboxes track relocation/deletion of each group. -> -> Owner: **keen-dove-772** (sequencing decision). Dissolution trigger: all rows resolved (every RELOCATE landed at its proposed home, every DELETE confirmed removed). - ---- - -## Network / locality - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `NetworkLocality` (+ `Loopback`, `Lan`, `Internet`, `WslNat`, `WslBridged`, `DockerBridge`, `TailscaleOverlay`) | `fleet_intent_network.dag`, `network_topology.dag`, `access_iam_validation_test.dag` | RELOCATE | `extdeps/network/topology.dag` or `product/network_topology.dag` | [ ] | -| `NetworkAddressability` | `fleet_intent_network.dag` | RELOCATE | Same as `NetworkLocality` | [ ] | -| `NetworkEgressClass` | `fleet_intent_network.dag` | RELOCATE | Same as `NetworkLocality` | [ ] | -| `NetworkInterface` | `fleet_intent_network.dag` | RELOCATE | Same as `NetworkLocality` | [ ] | - ---- - -## Isolation / container - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `IsolationBoundary` (+ `SharedHostHome`, `PerJobFilesystem`, `ContainerHermetic`, `Vm`) | `fleet_intent.dag`, `fleet_container.dag`, `ci_fleet.dag` | RELOCATE | `product/fleet_container.dag` or `extdeps/container/types.dag` | [ ] | -| `ContainerRuntime` (`Docker`, `Podman`, `Gvisor`, `Ubicloud`, `GcloudRun`) | `fleet_intent.dag`, `fleet_container.dag` | RELOCATE | `extdeps/container/types.dag` | [ ] | -| `PersistenceKind` (`Ephemeral`, `Persistent`, `CachedMirror`) | `fleet_intent.dag` (via `StorageDevice`) | RELOCATE | `product/fleet_intent.dag` or `extdeps/storage/types.dag` | [ ] | - ---- - -## Storage - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `StorageDevice` (+ `NvmeDevice`, `BareDevice` variants) | `fleet_intent.dag`, `fleet_intent_storage_witness_test.dag` | RELOCATE | `extdeps/storage/types.dag` (catalog rows already there) | [ ] | -| `StorageMedium` (`Nvme`, `Ssd`, `Hdd`, `NetworkAttached`, `EphemeralFs`) | `fleet_intent.dag` | RELOCATE | `extdeps/storage/types.dag` | [ ] | -| `StorageMount` | `fleet_intent.dag` | RELOCATE | `product/fleet_intent.dag` | [ ] | -| `storage_device_capacity`, `storage_device_medium`, `storage_device_read_bandwidth`, `storage_device_write_bandwidth`, `storage_device_pcie_link`, `storage_device_serial`, `nvme_storage_device` | `fleet_intent.dag`, `fleet_intent_storage_witness_test.dag` | RELOCATE | Co-locate with `StorageDevice` | [ ] | - ---- - -## Memory / GPU / accelerator - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `MemoryKind` (`Dram`, `Hbm`, `UnifiedShared`) | `ci_fleet.dag` | RELOCATE | `extdeps/memory/types.dag` | [ ] | -| `MemoryFacts`, `MemoryDevice` | `ci_fleet.dag`, `fleet_intent.dag` | RELOCATE | `extdeps/memory/types.dag` | [ ] | -| `GpuFacts`, `GpuComputeCapability`, `GpuRuntime`, `gpu_compute_capability_sm_label` | `hardware_selection.dag` | RELOCATE | `extdeps/gpu/types.dag` | [ ] | -| `AcceleratorFacts` | `hardware_selection.dag` | RELOCATE | `extdeps/gpu/types.dag` | [ ] | -| `LatencyClass` | `fleet_intent.dag` | RELOCATE | `extdeps/network/topology.dag` or `std/measure` | [ ] | - ---- - -## Money / cost - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `MoneyMicros`, `money_micros`, `money_micros_count` | `hardware_selection.dag` | RELOCATE | `std/measure.dag` (`MoneyAmount` alias already there; promote constructor/accessor) | [ ] | -| `CostClass` (`OwnedMarginalZero`, `PerSecondBilled`, `Metered`, `DeveloperAttention`) | `ci_fleet.dag` | RELOCATE | `product/ci_fleet.dag` or `extdeps/provider/cost.dag` | [ ] | -| `ProviderIdentity` | `ci_fleet.dag`, `ci_budget_tree.dag`, `runner_spec_from_offer.dag` | RELOCATE | `product/ci_fleet.dag` or `extdeps/provider/identity.dag` | [ ] | -| `CostModel`, `ProviderCostModel`, `CostEstimate`, `AvailabilityWindow` | `ci_fleet.dag` | RELOCATE | `product/ci_fleet.dag` | [ ] | - ---- - -## Compute host / offer / supply - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `ProcessorKind` (`CpuProcessor`, `GpuProcessor`, `AcceleratorProcessor`) | `ci_fleet.dag`, `fleet_intent.dag` | RELOCATE | `extdeps/cpu/types.dag` | [ ] | -| `ExecutionSurface` | `ci_fleet.dag`, `fleet_intent.dag`, `host_effect.dag` | RELOCATE | `product/fleet_intent.dag` | [ ] | -| `ComputeHost` | `host_effect.dag`, `host_effect_realize.dag`, `ci_fleet.dag`, `fleet_intent.dag`, `ci_runner_placement.dag`, `fleet_host_budget.dag`, `runner_spec_from_offer.dag`, `ci_budget_tree.dag` | RELOCATE | `product/fleet_intent.dag` (the host-model authority) | [ ] | -| `compute_host_primary_cpu`, `compute_host_ram_bytes_total` | `rust_gates_ci.dag`, `ci_runner_placement.dag`, `fleet_host_budget.dag`, `ci_budget_tree.dag`, `runner_spec_from_offer.dag` | RELOCATE | Co-locate with `ComputeHost` | [ ] | -| `ComputeSupplyFacts` | `ci_fleet.dag`, `fleet_intent.dag` | RELOCATE | `product/ci_fleet.dag` | [ ] | -| `ComputeOffer` | `ci_fleet.dag`, `fleet_intent.dag`, `ci_runner_placement.dag`, `fleet_host_budget.dag`, `runner_spec_from_offer.dag`, `ci_budget_tree.dag` | RELOCATE | `product/ci_fleet.dag` (the offer authority) | [ ] | -| `placement_supply_row` | `ci_runner_placement.dag`, `fleet_host_budget.dag`, `ci_budget_tree.dag`, `runner_spec_from_offer.dag` | RELOCATE | `product/placement_supply.dag` (already exists as `PlacementSupplyRow`) | [ ] | - ---- - -## Resource envelope / requirements - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `ResourceEnvelope` | `fleet_container.dag`, `src/v2/workflow/ci_floor_plan.dag`, `src/v1/stage0/inert_carrier_project.rs` | RELOCATE | `product/fleet_container.dag` or `std` (scheduler authority) | [ ] | -| `CpuRequirement`, `GpuRequirement`, `MemoryRequirement`, `StorageRequirement`, `NetworkRequirement` | `fleet_container.dag`, `src/v2/workflow/ci_floor_plan.dag` | RELOCATE | Co-locate with `ResourceEnvelope` | [ ] | -| `ToolchainCapability`, `ToolchainRequirement`, `ToolchainEnvIsolation`, `BuildWrapperKind` | `fleet_intent.dag`, `fleet_container.dag`, `ci_fleet.dag` | RELOCATE | `extdeps/toolchain/types.dag` (types already there) | [ ] | - ---- - -## Work demand / parallelism - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `WorkDemand`, `ParallelismShape`, `ReducerLaws` | `src/v2/workflow/ci_floor_plan.dag`, `dsl/gunbc/ci_floor_measurement.dag`, `src/v1/stage0/inert_carrier_project.rs` | RELOCATE | `std/realization_schedule.dag` or `product/placement_supply.dag` | [ ] | -| `WorkUnitId`, `WorkUnit` | `execution_receipt_digest_test.dag`, `realization_measurement_keystone_test.dag` | RELOCATE | `std/realization_schedule.dag` | [ ] | - ---- - -## Admission / input envelope - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `InputEnvelope`, `InputBound`, `InputSizeAxis`, `AdmissionVerdict` | `input_envelope_admission_test.dag`, `gunbc_ci_corpus_envelope.dag` | RELOCATE | `product/ci_floor_measurement.dag` or `std/realization_schedule.dag` | [ ] | -| `input_admitted`, `gunbc_ci_corpus_envelope`, `parallel_run_demand_envelope`, `parallel_run_demand_envelope_shard_count` | `input_envelope_admission_test.dag`, `ci_floor_measurement.dag` | RELOCATE | Co-locate with `InputEnvelope` | [ ] | - ---- - -## Performance / execution receipts - -| Symbol | External consumers | Disposition | Proposed home | Status | -|--------|-------------------|-------------|---------------|--------| -| `PerformanceReceipt`, `ExecutionReceipt`, `CostReceipt`, `PricingSource`, `AmortizationScope` | `execution_receipt_digest_test.dag`, `realization_measurement_keystone_test.dag` | RELOCATE | `std/realization_measurement.dag` (receipt accessors already there) | [ ] | -| `execution_receipt_subject_digest`, `execution_receipt_performance_digest`, `execution_receipt_cost_digest`, `execution_receipt_full_digest` | `execution_receipt_digest_test.dag` | RELOCATE | Co-locate with `ExecutionReceipt` | [ ] | -| `performance_receipt_host_single_sample`, `cost_account_from_performance_receipts`, `performance_receipts_total_time` | `realization_measurement_keystone_test.dag` | RELOCATE | `std/realization_measurement.dag` | [ ] | -| `MeasurementConfidence` | `realization_measurement_keystone_test.dag` | RELOCATE | `std/realization_measurement.dag` | [ ] | - ---- - -## Fabric-internal stubs (DELETE) - -No external consumers confirmed. Safe to drop once #5889 lands. - -| Symbol | Disposition | Status | -|--------|-------------|--------| -| `ComputeWitness`, `Outcome`, `UpsertInputRef`, `ChangeSet`, `ArtifactRef`, `ArtifactSpec`, `WorkAction`, `WorkGraph`, `Partitioner`, `Reducer`, `EffectBoundary`, `SymbolicCost`, `Associativity`, `Commutativity`, `Idempotency`, `IdentityElement`, `InstructionSet` | DELETE (opaque Node stubs, zero consumers) | [ ] | -| `ComputeLease`, `ComputeLeaseWitness`, `ComputeLeaseEligibility`, `AllocationReceipt` | DELETE (lease machinery, zero external consumers) | [ ] | -| `MachineView`, `MissingDemandFact`, `DemandDimension` | DELETE (old matcher internals) | [ ] | -| `OomSignalClass`, `ProviderConstraint`, `ExecutionBudget`, `WatchdogLimit` | DELETE (no external consumers found in tree scan) | [ ] | -| `ComputeArtifactLocality` | DELETE (no external consumers found) | [ ] | -| `satisfies`, `isolation_satisfies`, `isolation_digest`, old `example_*` data, old witnesses | DELETE (replaced by connector model) | [x] (landed in #5889) | diff --git a/dsl/extdeps/languages/bash/program.dag b/dsl/extdeps/languages/bash/program.dag index 5a48cf15014..e8d7700683a 100644 --- a/dsl/extdeps/languages/bash/program.dag +++ b/dsl/extdeps/languages/bash/program.dag @@ -39,7 +39,6 @@ type ShellStmt | Pipe { left: ShellStmt, right: ShellStmt } | EnvPrefixed { name: String, value: ShellWord, command: ShellStmt } - | EnvUnsetCmd { name: String, command: ShellStmt } | WithRedir { stmt: ShellStmt, redir: Redir } | Heredoc { path: ShellWord, delimiter: String, body: String } @@ -137,8 +136,6 @@ fn serialize_stmt(s: ShellStmt) -> String { concat(serialize_stmt(s: l), concat(" | ", serialize_stmt(s: r))) EnvPrefixed { name: n, value: v, command: c } => concat(n, concat("=", concat(serialize_word(w: v), concat(" ", serialize_stmt(s: c))))) - EnvUnsetCmd { name: n, command: c } => - concat("env -u ", concat(n, concat(" ", serialize_stmt(s: c)))) WithRedir { stmt: st, redir: rd } => concat(serialize_stmt(s: st), concat(" ", serialize_redir(r: rd))) Exit { code: c } => concat("exit ", to_string(c)) diff --git a/dsl/gunbc/roadmap_authority.dag b/dsl/gunbc/roadmap_authority.dag index 888f03d47cd..aa77da4d93a 100644 --- a/dsl/gunbc/roadmap_authority.dag +++ b/dsl/gunbc/roadmap_authority.dag @@ -181,7 +181,6 @@ fn section_1() -> RoadmapSection { authored(id: "1-g2-runner", done: false, content: "**G2 runner deployment + cross-host placement** *(dispatched proud-tern-439: model + drift-gate)* — derive runners/host + registration from `fleet_intent`+envelope; **host-apply RESOLVED (operator-set, CI-humming plan authority): the gunbc-EMITTED converge shell** (`.github/fleet-converge.sh` = a §2 host-effect handler inhabiting the §6 carrier *by emission* — the emitted shell IS the executing consumer, escaping the §5 spec-without-execution trap; ctrl thin-runs it + parses a fail-closed receipt; §7 seed-shrink). In progress: emit (wise-eagle-664) + thin consume/run (fierce-carp)"), authored(id: "1-g3-caps", done: false, content: "**G3 cgroup caps** — `TasksMax`/`MemoryMax` host-set by hand, only *read* live; derive + reconcile via the **emitted converge shell + fail-closed receipt** (same carrier-inhabitation-by-emission as G2) so a hand-edit reds"), authored_doc(id: "1-ci-fabric", done: false, content: "**CI on compute fabric** — derive every host knob from one measured `ResourceEnvelope`; ends the crash-or-idle swing", path: "docs/plans/compute-envelope-model.md"), - authored_doc(id: "1-fabric-connector-migration", done: false, content: "**compute fabric connector migration** — relocate ~50 removed vocabulary symbols to their single-authority homes before #5889 can merge", path: "docs/plans/compute-fabric-connector-migration.md"), ], edges: [], ), @@ -327,7 +326,7 @@ fn section_5() -> RoadmapSection { section_prose(content: "Anchor (do not flip-flop): `.dag` = truth; purely self-hosting (v2 emits its own seed, no stage0 hand-edits); emit Rust + TypeScript; then shrink the seed to zero. → [plan](docs/plans/v2-self-hosting.md) · [de-fork audit](docs/plans/dsl-v2-defork-audit.md)"), section_prose(content: "Adjacent lane — algorithmic-cost rewrite engine (the §3 construction design; post-stability, natural once `.dag` is the self-hosted truth). → [plan](docs/plans/algebraic-rewrite-optimization.md)"), - section_prose(content: "**◆ Milestones (critical path):** front-end ✓ · emit-rust well-typed ✓ · de-fork Step 1 ✓ (#5473) · class-3 corpus-coherence + cargo-green seed ✓ (#5481) → **▸ NOW: emitted crate cargo-builds green** → std forks collapsed → real fixed point → **KEYSTONE: `regen --verify` in CI** → seed-honesty → **TERMINAL: `src/v1` deleted** *(§7 regen-fixpoint deferred, #5514; src/v1 NOT yet deletable)*"), + section_prose(content: "**◆ Milestones (critical path):** front-end ✓ · emit-rust well-typed ✓ · de-fork Step 1 ✓ (#5473) · class-3 corpus-coherence + cargo-green seed ✓ (#5481) → emitted crate cargo-builds green ✓ (#5777/#5873) → **▸ NOW: real fixed point** → `regen --verify` in CI ✓ (#5873) → seed-honesty → **TERMINAL: `src/v1` deleted** *(§7 regen-fixpoint deferred, #5514; src/v1 NOT yet deletable)*"), section_group( label: "", nodes: [ @@ -342,7 +341,7 @@ fn section_5() -> RoadmapSection { authored(id: "5-root-a", done: true, content: "**Root A — emit-seam grounding ✓ LANDED (#5734)**: String/`FreeMonoid`→host `Vec` via the #5428 `RustCorpusRepr` seam; drove HostNative cargo-green *(jolly-cat)*"), authored(id: "5-root-b", done: false, content: "**Root B — generic-inference keystone ✓ MERGED (#5552)** → def-unification (coproduct authority + aliases) → repoints (algebra/nat/integer/float/logic/effects/verification) → 🟡-marker dissolution *(bright-deer-111)*"), authored(id: "5-v1coupled", done: false, content: "v1-coupled `coercion`/`node` renames — deferred to v1-delete"), - authored(id: "5-cargo-green", done: false, content: "**emitted crate `cargo build`s green (Route-A last mile)** — execution-grounded, **re-measured after unmask** (#5714, deep-wolf, `--emit-fresh` full 108-emitted + 22-periphery closure): the earlier **38 E0308 was an E0433-masked LOWER BOUND** — true ceiling **647 E0308** (the §5-honesty call to re-measure-after-unmask was right, ~17x over the masked bound; *no* green was ever claimed against `38→0`). **Family A (24 map-literal `&str` keys) CLEARED → 0** (type-DERIVED key coercion, landed via the merged Lane-A re-baseline #5716/#5718 after the standalone #5714 was superseded/closed). **Pole = the Optional-on-String-carrier family: 638/647 (98.6%)** — and execution **REFUTED the first-cut 'pipe-first consumer-unwrap' root cause**: the dominant 638 are **signature-rendering** bugs (`expected String, found Option`), the Rust emitter **dropping the `Optional` wrapper on String text-carrier signature/field types** (the carrier path early-returns a bare `String` before the Optional-template renderer; `Int?` rendered `Option` correctly, only `String?` regressed). The function *body* already emits `Some/None` — only the *type* was unfaithful, so the model was right (`first` IS modeled `OptionalOf`, algebra.dag:295). Fix = **faithful Optional RENDERING** (`rust_carrier_optional_wrap`: wrap iff `return_cardinality == CardOptional`, cardinality-gated + a non-optional-stays-bare control) — #5716, **CONSTRUCTION not a coercion**. Trajectory by `--emit-fresh`: **647 → 9** (the 11-site renderer wrap) **→ 6** (3 fail-closed unwraps at the genuine `T?`-into-`T`-slot model-hole consumers — semver_compare_identifier, schedule_witness_entry_eq, runnable_eq — located `.expect`, never `unwrap_or_default`, the empty-None PANIC proven by catch_unwind against a fabricates-0 control). **E0308 POLE CLEARED: 647 → 0** (the final 6 — fn-item-as-value 1 @ semver.rs:226 · and-str 3 · Box/Rc DescentEvidence 2 — clear in deep-wolf's 6-tail PR; fn-item via a load-bearing record-collapse-fold consistency fix, gated). But **0 E0308 is a MILESTONE, not cargo-green** — execution **REFUTED the '0 E0308 = done' finish-line**: the lane measured E0308-ONLY (its brief said 'categorized E0308 count'), and the FULL cargo build shows **~32 OTHER pre-existing errors never counted**, constant across measures (orthogonal, not unmasked — 41=9+32, 38=6+32, 32=0+32). Re-census after the measure-tower pole CLEARED: **measure-Q LANDED (#5735)** — value-variant type-arg collapse, the E0560×12/E0614×7 `std_measure.rs` root — and **std_serialization C0 cleared (#5761)** (a fixed-point prereq). Full `--emit-fresh` cargo build now **5 TOTAL errors**; #5777 (open, merry-gull — E0599 `Clone`-bound + E0624 Optional-receiver) takes **5 → 2**, and the residual **2 are E0091×2** — one **Compose decision** away. So **cargo-green = 0 TOTAL cargo errors** (the real self-host build), no longer the measure pole. Model hole stays flagged: pipe-first is modeled `OptionalOf`, so a **model-forces-unwrap follow-up is owed UPSTREAM** (the Optional/`Value::Null` runway). **\"done\" = the emitted crate cargo-builds with 0 TOTAL errors (the E0308 milestone + the 32 cleared) + the PRs merged** — measure full-cargo, never a grep of one error family. *(jolly-cat #5705/#5716/#5718 + the 6-tail/32-baseline; supersedes the stale 38-lower-bound, the ~12,600 / ~150-latent estimate, the refuted pipe-first-unwrap root cause, AND the refuted '0 E0308 = cargo-green' finish-line)*"), + authored(id: "5-cargo-green", done: true, content: "**emitted crate `cargo build`s green (Route-A last mile) ✓ LANDED (#5777/#5873)** — execution-grounded **re-census CONFIRMED 0 TOTAL errors** (cool-hawk-908: `regen_stage0 --emit-fresh` → `cargo build` debug+release, 2026-06-28). **REFUTED** the stale 'residual 2 = E0091×2' hypothesis: #5777 cleared E0599/E0624 **and** E0091 (`alias_unused_param_names` → `PhantomData` for unused type-alias params; `type_alias_phantom_param_test` witnesses). Trajectory: 647 E0308 → 0 (#5716/#5718/#5717) → ~32 other-family → 5 TOTAL → **0 TOTAL** (#5777). #5873: two-generation regen cutover + `regen --verify` CI gate. Compose emits `PhantomData<(Algebra, MachineConstraint)>`. Model hole stays flagged: pipe-first `OptionalOf` unwrap follow-up owed upstream (`Value::Null` runway). *(jolly-cat #5705/#5716/#5718 + cool-hawk-908 re-census)*"), authored(id: "5-real-fixpoint", done: false, content: "real fixed point: `content_hash` stage1==stage2 (dissolve placeholder hashes)"), authored(id: "5-regen-verify", done: true, content: "wire `regen_stage0 --verify` lockstep gate into CI — enforces no hand-edits to **generated** stage0 files (`GENERATED_STAGE0_FILES`); `HAND_MAINTAINED_STAGE0_FILES` are copied through, excluded by design ← **keystone**"), authored(id: "5-dissolve-patches", done: false, content: "dissolve seed hand-patches (`patch_*` / `HAND_MAINTAINED_STAGE0_FILES`) — the `emit_rust` hand-sync caveat the gate reproduces (the required-facts plan is dissolved into the executable `RegenVerifyGate`)"), diff --git a/dsl/product/compute_fabric.dag b/dsl/product/compute_fabric.dag index 75c0a842107..80a4809a6b1 100644 --- a/dsl/product/compute_fabric.dag +++ b/dsl/product/compute_fabric.dag @@ -1,134 +1,1359 @@ module product.compute_fabric -import std.types { NonEmptyStr, Bool, List } -import std.measure { HardwareThreadCount, hardware_thread_count, hardware_thread_count_value, measure_le } +import std.nat { Nat } +import std.types { NonEmptyStr, ContentHash, Bool, Int, LogicalTime, List } +import extdeps.vendor { Vendor } +import extdeps.hardware { Hardware } +import extdeps.vendor.nvidia { nvidia } +import extdeps.vendor.amd { amd } +import extdeps.vendor.ampere { ampere } +import std.os.types { + FileSystemSemantics, + Linux, + OperatingSystemSurface, + Posix, + PosixProcess, + ProcessSemantics, +} +import std.float { Float } +import std.measure { + Measure, + Time, + Currency, + Micro, + One, + Nano, + ByteSize, + MoneyAmount, + HardwareThreadCount, + Count, + byte_size, + byte_size_count, + measure_count, + hardware_thread_count, + hardware_thread_count_value, + measure_scale_fraction_ceil, + hertz, + watt, + Bandwidth, +} +import std.currency { CurrencyCode } +import std.disposition { Disposition, Scaffold, SingleAuthority } +import std.decl_ref { DeclarationRef, WholeDeclaration } +import std.verification { NanosecondDuration, nanosecond_duration, nanosecond_duration_count } +import std.realization_schedule { CostAccount } +import std.realization_measurement { cost_account_measured_from_time, cost_account_measured_from_time_and_space } +import std.content_hash { + content_hash_atom, + content_hash_combine, + content_hash_tagged, +} +import std.cache_identity { CacheInterfaceId, ArtifactKindId, ExecutionReceiptDigest } +import extdeps.cpu { + CpuFacts, + cpu_facts_architecture, + cpu_facts_sustained_per_thread_hz, + cpu_facts_threads, +} +import extdeps.cpu.types { + CpuDeploymentFacts, + CpuModelCatalogRow, + CpuSocket, + SocketPackage, +} +import extdeps.storage.types { NvmeDeviceCatalogRow, PcieLink } +import extdeps.ctrl.jobserver { CtrlJobserverConfig } +import extdeps.toolchain.types { + Architecture, + Aarch64, + X86_64, + Armv7, + Riscv64, + Wasm32, +} +import product.placement_supply { HostIdentity, PlacementSupplyRow } +import std.baseboard { ServerBaseboard } +import std.realization_width { bounded_host_spawn_width, memory_aware_spawn_width, int_min } + +type ComputeWitness = Node +type Outcome = Node + +type UpsertInputRef = Node +type ChangeSet = Node +type ArtifactRef = Node +type ArtifactSpec = Node +type WorkAction = Node +type WorkGraph = Node +type Partitioner = Node +type Reducer = Node +type EffectBoundary = Node +type SymbolicCost = Node +type Associativity = Node +type Commutativity = Node +type Idempotency = Node +type IdentityElement = Node +type InstructionSet = Node + +type NetworkLocality = + Loopback + | Lan + | Internet + | WslNat + | WslBridged + | DockerBridge + | TailscaleOverlay +type NetworkAddressability = Unicast | Multicast | Broadcast +type NetworkEgressClass = + None + | CratesIo + | InternalMirror + | Unrestricted +type IsolationBoundary = SharedHostHome | PerJobFilesystem | ContainerHermetic | Vm +type PersistenceKind = Ephemeral | Persistent | CachedMirror +type MemoryKind = Dram | Hbm | UnifiedShared +type StorageMedium = Nvme | Ssd | Hdd | NetworkAttached | EphemeralFs +type LatencyClass = UltraLow | Low | Medium | High +type GpuRuntime = Cuda | Rocm | Metal | OpenCl +type ContainerRuntime = Docker | Podman | Gvisor | Ubicloud | GcloudRun +type CostClass = OwnedMarginalZero | PerSecondBilled | Metered | DeveloperAttention +type OomSignalClass = KillProcess | Throttle | ReportOnly + +type Duration = Measure + +type MoneyMicros = MoneyAmount + +fn money_micros(count: Nat) -> MoneyMicros { + MoneyMicros { count: count } +} + +fn money_micros_count(m: MoneyMicros) -> Nat { + measure_count(m: m) +} + +type ProviderIdentity = NonEmptyStr where brand("ProviderIdentity") + +type ComputeArtifactLocality + = InProcess + | PerRunnerColocation + | PerHostColocation + | CrossHostFetch + +type StorageMount { + mount_path: NonEmptyStr + device: StorageDevice + lifecycle: PersistenceKind +} + +type MemoryFacts { capacity: ByteSize, memory_kind: MemoryKind } + +type GpuComputeCapability { + vendor: Vendor + major: Int + minor: Int +} + +type UnsubstantiatedGpuCapabilityLabel { vendor: Vendor } -type Fabric = NonEmptyStr where brand("Fabric") +type GpuCapabilityLabelOutcome + = GpuCapabilityLabelOk { label: String } + | GpuCapabilityLabelRejected { reason: UnsubstantiatedGpuCapabilityLabel } + +fn gpu_compute_capability_sm_label(cap: GpuComputeCapability) -> GpuCapabilityLabelOutcome { + if cap.vendor == nvidia { + GpuCapabilityLabelOk { + label: concat("sm_", concat(to_string(cap.major), concat("_", to_string(cap.minor)))) + } + } else { + if cap.vendor == amd { + GpuCapabilityLabelOk { + label: concat("gfx", concat(to_string(cap.major), to_string(cap.minor))) + } + } else { + GpuCapabilityLabelRejected { + reason: UnsubstantiatedGpuCapabilityLabel { vendor: cap.vendor } + } + } + } +} -type HardRequirements { threads: HardwareThreadCount } +type GpuFacts { + vendor: Vendor + model: NonEmptyStr + compute_capability: GpuComputeCapability? + memory: MemoryFacts + supported_runtimes: List +} -type Shape { hard: HardRequirements } +type AcceleratorFacts { vendor: Vendor, model: NonEmptyStr } -fn shape_covers(offered: Shape, needed: Shape) -> Bool { - measure_le(a: needed.hard.threads, b: offered.hard.threads) +type MemoryDevice { + capacity: ByteSize + memory_kind: MemoryKind + bandwidth: Bandwidth? } -type Program { - id: NonEmptyStr - hard: HardRequirements +type StorageDevice + = NvmeDevice { + catalog: NvmeDeviceCatalogRow + serial: NonEmptyStr? + persistence: PersistenceKind + } + | BareDevice { + capacity: ByteSize + medium: StorageMedium + serial: NonEmptyStr? + read_bandwidth: Bandwidth? + write_bandwidth: Bandwidth? + persistence: PersistenceKind + } + +fn nvme_storage_device(catalog: NvmeDeviceCatalogRow, persistence: PersistenceKind) -> StorageDevice { + NvmeDevice { + catalog: catalog, + serial: none, + persistence: persistence, + } } -fn derive_shape(p: Program) -> Shape { - Shape { hard: p.hard } +fn storage_device_capacity(d: StorageDevice) -> ByteSize { + match d { + NvmeDevice { catalog: c, serial: _, persistence: _ } => c.capacity_bytes + BareDevice { capacity: cap, medium: _, serial: _, read_bandwidth: _, write_bandwidth: _, persistence: _ } => cap + } } -type ComputeNeed { - fabric: Fabric - program: Program +fn storage_device_medium(d: StorageDevice) -> StorageMedium { + match d { + NvmeDevice { catalog: _, serial: _, persistence: _ } => Nvme + BareDevice { capacity: _, medium: m, serial: _, read_bandwidth: _, write_bandwidth: _, persistence: _ } => m + } } -type Opportunity { - fabric: Fabric - offers: Shape - tag: NonEmptyStr +fn storage_device_read_bandwidth(d: StorageDevice) -> Bandwidth? { + match d { + NvmeDevice { catalog: c, serial: _, persistence: _ } => Present { value: c.sequential_read } + BareDevice { capacity: _, medium: _, serial: _, read_bandwidth: bw, write_bandwidth: _, persistence: _ } => bw + } } -type Connection - = Bound { opportunity: Opportunity } - | Pending - | Unmet { reason: NonEmptyStr } +fn storage_device_write_bandwidth(d: StorageDevice) -> Bandwidth? { + match d { + NvmeDevice { catalog: c, serial: _, persistence: _ } => Present { value: c.sequential_write } + BareDevice { capacity: _, medium: _, serial: _, read_bandwidth: _, write_bandwidth: bw, persistence: _ } => bw + } +} -fn connect(need: ComputeNeed, opportunities: List) -> Connection { - let shape = derive_shape(p: need.program) +fn storage_device_pcie_link(d: StorageDevice) -> PcieLink? { + match d { + NvmeDevice { catalog: c, serial: _, persistence: _ } => Present { value: c.pcie_link } + BareDevice { capacity: _, medium: _, serial: _, read_bandwidth: _, write_bandwidth: _, persistence: _ } => none + } +} + +fn storage_device_serial(d: StorageDevice) -> NonEmptyStr? { + match d { + NvmeDevice { catalog: _, serial: s, persistence: _ } => s + BareDevice { capacity: _, medium: _, serial: s, read_bandwidth: _, write_bandwidth: _, persistence: _ } => s + } +} + +fn storage_device_persistence(d: StorageDevice) -> PersistenceKind { + match d { + NvmeDevice { catalog: _, serial: _, persistence: p } => p + BareDevice { capacity: _, medium: _, serial: _, read_bandwidth: _, write_bandwidth: _, persistence: p } => p + } +} + +type NetworkInterface { + addressability: NetworkAddressability + bandwidth: Bandwidth? + latency_class: LatencyClass? + locality: NetworkLocality +} + +type ExecutionSurface { + os: OperatingSystemSurface + isolation: IsolationBoundary + container_runtime: ContainerRuntime? + toolchains: List + mounted_storage: List + network: List +} + +type ProcessorKind + = CpuProcessor { cpu: CpuFacts } + | GpuProcessor { gpu: GpuFacts } + | AcceleratorProcessor { accelerator: AcceleratorFacts } + +type ComputeHost { + identity: HostIdentity + baseboard: ServerBaseboard? + processors: List + memory: List + storage: List + network_interfaces: List +} + +type ProcessorCostModel { placeholder: Bool } +type ProviderCostModel { placeholder: Bool } + +type CostModel { + cost_class: CostClass + rates: ProviderCostModel +} + +type ComputeSupplyFacts { + physical: ComputeHost + execution: ExecutionSurface + cost: CostModel? + observed_performance: List +} + +type ProviderConstraint + = HostJobserverFifo { fifo_path: NonEmptyStr, config_ref: CtrlJobserverConfig } + | SharedHomeRoot { path: NonEmptyStr } + | MaxConcurrentRunners { cap: Int } + | OomBehavior { signal: OomSignalClass } + +type AvailabilityWindow { open: LogicalTime, close: LogicalTime? } +type CostEstimate { model: CostModel, unit_price: MoneyMicros, currency: CurrencyCode } + +fn per_second_billed_cost_quote(unit_price: MoneyMicros, currency: CurrencyCode) -> CostEstimate { + CostEstimate { + model: CostModel { + cost_class: PerSecondBilled, + rates: ProviderCostModel { placeholder: true }, + }, + unit_price: unit_price, + currency: currency, + } +} + +fn cost_estimate_unit_price_micros(quote: CostEstimate) -> Nat { + money_micros_count(m: quote.unit_price) +} + +type ComputeOffer { + provider: ProviderIdentity + supply: ComputeSupplyFacts + available_window: AvailabilityWindow + cost_quote: CostEstimate? + constraints: List +} + +type MachineView { label: NonEmptyStr, host: HostIdentity } + +fn machine_view(host: ComputeHost, surface: ExecutionSurface) -> MachineView { + MachineView { + label: machine_view_label(host: host, surface: surface), + host: host.identity, + } +} + +fn machine_view_label(host: ComputeHost, surface: ExecutionSurface) -> NonEmptyStr { + match surface.os.distro_or_product { + Present { value: distro } => distro + Absent => host.identity + } +} + +fn placement_supply_row(identity: HostIdentity, cpu: CpuFacts, ram_bytes_total: ByteSize) -> PlacementSupplyRow { + PlacementSupplyRow { + identity: identity, + hardware_threads: cpu_facts_threads(cpu), + clock_hz: cpu_facts_sustained_per_thread_hz(cpu), + ram_bytes: ram_bytes_total, + } +} + +fn compute_host_primary_cpu(host: ComputeHost) -> CpuFacts? { + match host.processors.first() { + Absent => none + Present { value: kind } => + match kind { + CpuProcessor { cpu: c } => Present { value: c } + GpuProcessor { gpu: _ } => none + AcceleratorProcessor { accelerator: _ } => none + } + } +} + +fn compute_host_ram_bytes_total(host: ComputeHost) -> ByteSize { + byte_size( + host.memory + |> fold( + init: 0, + f: (acc, dev) => acc + byte_size_count(dev.capacity), + ), + ) +} + +fn offer_placement_supply_row(offer: ComputeOffer) -> PlacementSupplyRow? { + match compute_host_primary_cpu(host: offer.supply.physical) { + Absent => none + Present { value: cpu } => + Present { + value: placement_supply_row( + identity: offer.supply.physical.identity, + cpu: cpu, + ram_bytes_total: compute_host_ram_bytes_total(host: offer.supply.physical), + ), + } + } +} + +fn hardware_threads_honor_demand(supply: HardwareThreadCount, demand: HardwareThreadCount) -> Bool { + hardware_thread_count_value(supply) >= hardware_thread_count_value(demand) +} + +fn cpu_threads_satisfies(supply: PlacementSupplyRow, demand: CpuRequirement?) -> Bool { + match demand { + Absent => true + Present { value: req } => + hardware_threads_honor_demand(supply: supply.hardware_threads, demand: req.min_threads) + } +} + +fn cpu_architecture_satisfies(host: ComputeHost, demand: CpuRequirement?) -> Bool { + match demand { + Absent => true + Present { value: req } => + match req.architecture { + Absent => true + Present { value: arch } => + match compute_host_primary_cpu(host: host) { + Absent => false + Present { value: cpu } => cpu_facts_architecture(cpu: cpu) == arch + } + } + } +} + +fn memory_bytes_honor_demand(supply: ByteSize, demand: ByteSize) -> Bool { + byte_size_count(supply) >= byte_size_count(demand) +} + +fn memory_requirement_satisfies(supply: PlacementSupplyRow, demand: MemoryRequirement?) -> Bool { + match demand { + Absent => true + Present { value: req } => + memory_bytes_honor_demand(supply: supply.ram_bytes, demand: req.min_bytes) + } +} + +fn requires_placement_facts(demand: WorkDemand) -> Bool { + match demand.resources.cpu { + Present { value: _ } => true + Absent => + match demand.resources.memory { + Present { value: _ } => true + Absent => false + } + } +} + +fn satisfied_dimensions_for(demand: WorkDemand) -> List { + let parallelism_dims = match demand.parallelism { + IndependentShards { shard_count: _ } => [DemandParallelism] + SingleWorkItem => [] + DependencyGraphParallel { graph: _ } => [] + PartitionedReduce { + partitioner: _, + map: _, + reduce: _, + laws: _, + } => [] + } + concat( + [DemandIsolation], + concat( + parallelism_dims, + concat( + match demand.resources.cpu { + Absent => [] + Present { value: _ } => [DemandCpu] + }, + match demand.resources.memory { + Absent => [] + Present { value: _ } => [DemandMemory] + }, + ), + ), + ) +} + +fn work_demand_shard_count(demand: WorkDemand) -> Int { + match demand.parallelism { + SingleWorkItem => 1 + IndependentShards { shard_count } => shard_count + DependencyGraphParallel { graph: _ } => 1 + PartitionedReduce { + partitioner: _, + map: _, + reduce: _, + laws: _, + } => 1 + } +} + +type ResourceEnvelope { + cpu: CpuRequirement? + gpu: GpuRequirement? + memory: MemoryRequirement? + storage: StorageRequirement? + network: NetworkRequirement? +} + +type CpuRequirement { min_threads: HardwareThreadCount, architecture: Architecture? } +type GpuRequirement { min_vram: ByteSize, runtimes: List } +type MemoryRequirement { min_bytes: ByteSize } +type StorageRequirement { min_bytes: ByteSize, persistence: PersistenceKind } +type NetworkRequirement { egress: NetworkEgressClass?, ambient_allowed: Bool } +type OperatingSystemRequirement { surface: OperatingSystemSurface } +type IsolationRequirement { boundary: IsolationBoundary } +type ToolchainRequirement { name: NonEmptyStr, version: NonEmptyStr } + +type ArtifactLocalityRequirement { + artifact_kind: ArtifactKindId + locality: ComputeArtifactLocality +} + +type ToolchainEnvIsolation + = SharedHomeAcrossJobs + | PerJobCargoHome { cargo_home_var: NonEmptyStr, rustup_home_var: NonEmptyStr } + | HermeticContainer + +type BuildWrapperKind = CtrlBuild | BakedInImage | NoBuildWrapper + +type ToolchainCapability { + name: NonEmptyStr + version: NonEmptyStr + env_isolation: ToolchainEnvIsolation + build_wrapper: BuildWrapperKind? + linked_cache_interface: CacheInterfaceId? +} + +type InputSizeAxis + = WitnessCount + | SourceNodeCount + | CorpusNodeCount + +type InputBound { + axis: InputSizeAxis + max: Measure +} + +type InputEnvelope + = BoundedInput { bounds: List } + | EnvelopeUnknown + +type AdmissionVerdict + = Admitted + | RefusedOverEnvelope { axis: InputSizeAxis } + | RefusedUndeclared + +fn find_input_bound_for_axis(declared: List, axis: InputSizeAxis) -> InputBound? { fold( - opportunities, - init: Unmet { reason: "no_matching_opportunity" }, - f: (acc, opp) => + declared, + init: none, + f: (acc, b) => match acc { - Bound { opportunity: _ } => acc - Pending => acc - Unmet { reason: _ } => - if opp.fabric == need.fabric && shape_covers(offered: opp.offers, needed: shape) { - Bound { opportunity: opp } - } else { - acc + Present { value: _ } => acc + Absent => if b.axis == axis { Present { value: b } } else { none } + } + ) +} + +fn input_admitted(envelope: InputEnvelope, actual: List) -> AdmissionVerdict { + match envelope { + EnvelopeUnknown => RefusedUndeclared + BoundedInput { bounds: declared } => + fold( + actual, + init: Admitted, + f: (verdict, actual_bound) => + match verdict { + RefusedOverEnvelope { axis: _ } => verdict + RefusedUndeclared => verdict + Admitted => + match find_input_bound_for_axis(declared: declared, axis: actual_bound.axis) { + Absent => RefusedOverEnvelope { axis: actual_bound.axis } + Present { value: decl_bound } => + if actual_bound.max.count > decl_bound.max.count { + RefusedOverEnvelope { axis: actual_bound.axis } + } else { + Admitted + } + } } + ) + } +} + +type WorkDemand { + resources: ResourceEnvelope + os: OperatingSystemRequirement? + isolation: IsolationRequirement + toolchains: List + parallelism: ParallelismShape + data_locality: List + effects: List + input_envelope: InputEnvelope +} + +fn projected_concurrent_peak(width: HardwareThreadCount, per_shard_peak: ByteSize) -> ByteSize { + measure_scale_fraction_ceil(m: per_shard_peak, num: hardware_thread_count_value(width), den: 1) +} + +fn parallel_run_demand_envelope(width: HardwareThreadCount, per_shard_peak: ByteSize) -> ResourceEnvelope { + ResourceEnvelope { + cpu: Present { value: CpuRequirement { min_threads: width, architecture: none } }, + gpu: none, + memory: Present { + value: MemoryRequirement { + min_bytes: projected_concurrent_peak(width: width, per_shard_peak: per_shard_peak) } + }, + storage: none, + network: none, + } +} + +fn parallel_run_demand_envelope_from_budget( + shard_count: Int, + hardware_threads: HardwareThreadCount, + memory_budget: ByteSize, + per_shard_peak: ByteSize, + conservative_fallback_width: Int +) -> ResourceEnvelope { + parallel_run_demand_envelope( + width: memory_aware_spawn_width( + shard_count: shard_count, + hardware_threads: hardware_threads, + memory_budget: memory_budget, + per_shard_peak: per_shard_peak, + conservative_fallback_width: conservative_fallback_width + ), + per_shard_peak: per_shard_peak ) } -data example_fabric: Fabric = "ci-fleet" +fn demand_envelope_memory(envelope: ResourceEnvelope) -> ByteSize { + match envelope.memory { + Absent => byte_size(0) + Present { value: req } => req.min_bytes + } +} -data example_small_program: Program = Program { - id: "small-prog", - hard: HardRequirements { threads: hardware_thread_count(4) }, +fn demand_envelope_thread_count(envelope: ResourceEnvelope) -> Nat { + match envelope.cpu { + Absent => 0 + Present { value: req } => hardware_thread_count_value(req.min_threads) + } } -data example_large_program: Program = Program { - id: "large-prog", - hard: HardRequirements { threads: hardware_thread_count(64) }, +fn demand_envelope_fits_budget(envelope: ResourceEnvelope, budget: ByteSize) -> Bool { + memory_bytes_honor_demand(supply: budget, demand: demand_envelope_memory(envelope: envelope)) } -data example_opportunity: Opportunity = Opportunity { - fabric: example_fabric, - offers: Shape { hard: HardRequirements { threads: hardware_thread_count(128) } }, - tag: "srv1", +type ParallelismShape + = SingleWorkItem + | IndependentShards { shard_count: Int } + | DependencyGraphParallel { graph: WorkGraph } + | PartitionedReduce { + partitioner: Partitioner + map: WorkAction + reduce: Reducer + laws: ReducerLaws + } + +type ReducerLaws { + associative: ComputeWitness + commutative: ComputeWitness? + identity: IdentityElement? + idempotent: ComputeWitness? } -data example_small_opportunity: Opportunity = Opportunity { - fabric: example_fabric, - offers: Shape { hard: HardRequirements { threads: hardware_thread_count(8) } }, - tag: "srv1", +type ExecutionBudget { + expected_cost: SymbolicCost + provider_model: ProviderCostModel + watchdog: WatchdogLimit +} + +type WatchdogLimit { max_wall: Duration } + +type DemandDimension + = DemandCpu + | DemandGpu + | DemandMemory + | DemandStorage + | DemandNetwork + | DemandOs + | DemandIsolation + | DemandToolchains + | DemandParallelism + | DemandDataLocality + | DemandEffects + +type MissingDemandFact { dimension: DemandDimension, required: NonEmptyStr } + +type ComputeLeaseWitness { + provider: ProviderIdentity + satisfied: List } -fn witness_need_is_only_fabric_plus_program() -> Bool { - let need = ComputeNeed { fabric: example_fabric, program: example_small_program } - hardware_thread_count_value(t: need.program.hard.threads) == 4 +type ComputeLeaseEligibility + = Eligible { witness: ComputeLeaseWitness } + | Rejected { reason: MissingDemandFact } + +type AllocationReceipt { + provider: ProviderIdentity + scope: NonEmptyStr + acquired_at: LogicalTime } -fn witness_shape_is_derived_from_program() -> Bool { - let small_shape = derive_shape(p: example_small_program) - let large_shape = derive_shape(p: example_large_program) - small_shape.hard.threads != large_shape.hard.threads +type ComputeLease { + offer: ComputeOffer + demand: WorkDemand + allocation: AllocationReceipt + eligibility: ComputeWitness } -fn witness_fabric_binds_need_to_opportunity() -> Bool { - let need = ComputeNeed { fabric: example_fabric, program: example_small_program } - match connect(need: need, opportunities: [example_opportunity]) { - Bound { opportunity: _ } => true - Pending => false - Unmet { reason: _ } => false +fn isolation_boundary_ordinal(boundary: IsolationBoundary) -> Int { + match boundary { + SharedHostHome => 0 + PerJobFilesystem => 1 + ContainerHermetic => 2 + Vm => 3 } } -fn witness_exec_and_ci_are_just_programs() -> Bool { - let exec_prog = Program { - id: "exec-cmd", - hard: HardRequirements { threads: hardware_thread_count(4) }, +fn isolation_boundary_label(boundary: IsolationBoundary) -> NonEmptyStr { + match boundary { + SharedHostHome => "Vm/ContainerHermetic/PerJobFilesystem/SharedHostHome" + PerJobFilesystem => "Vm/ContainerHermetic/PerJobFilesystem" + ContainerHermetic => "Vm/ContainerHermetic" + Vm => "Vm" } - let ci_job = Program { - id: "ci-job", - hard: HardRequirements { threads: hardware_thread_count(4) }, +} + +fn isolation_satisfies(supply: IsolationBoundary, demand: IsolationBoundary) -> Bool { + isolation_boundary_ordinal(boundary: supply) >= isolation_boundary_ordinal(boundary: demand) +} + +fn satisfies(offer: ComputeOffer, demand: WorkDemand) -> ComputeLeaseEligibility { + if !isolation_satisfies(supply: offer.supply.execution.isolation, demand: demand.isolation.boundary) { + Rejected { + reason: MissingDemandFact { + dimension: DemandIsolation, + required: isolation_boundary_label(boundary: demand.isolation.boundary), + }, + } + } else if !requires_placement_facts(demand: demand) { + Eligible { + witness: ComputeLeaseWitness { + provider: offer.provider, + satisfied: [DemandIsolation], + }, + } + } else { + match offer_placement_supply_row(offer: offer) { + Absent => + Rejected { + reason: MissingDemandFact { + dimension: DemandCpu, + required: "primary_cpu", + }, + } + Present { value: row } => + if !cpu_threads_satisfies(supply: row, demand: demand.resources.cpu) { + Rejected { + reason: MissingDemandFact { + dimension: DemandCpu, + required: "min_threads", + }, + } + } else if !cpu_architecture_satisfies(host: offer.supply.physical, demand: demand.resources.cpu) { + Rejected { + reason: MissingDemandFact { + dimension: DemandCpu, + required: "cpu_architecture", + }, + } + } else if !memory_requirement_satisfies(supply: row, demand: demand.resources.memory) { + Rejected { + reason: MissingDemandFact { + dimension: DemandMemory, + required: "min_bytes", + }, + } + } else { + Eligible { + witness: ComputeLeaseWitness { + provider: offer.provider, + satisfied: satisfied_dimensions_for(demand: demand), + }, + } + } + } } - let need_exec = ComputeNeed { fabric: example_fabric, program: exec_prog } - let need_ci = ComputeNeed { fabric: example_fabric, program: ci_job } - match connect(need: need_exec, opportunities: [example_opportunity]) { - Bound { opportunity: _ } => - match connect(need: need_ci, opportunities: [example_opportunity]) { - Bound { opportunity: _ } => true - Pending => false - Unmet { reason: _ } => false - } - Pending => false - Unmet { reason: _ } => false +} + +type WorkUnitId = NonEmptyStr where brand("WorkUnitId") + +type WorkUnit { + id: WorkUnitId + inputs: List> + action: WorkAction + output: ArtifactSpec + requirements: WorkDemand +} + +type MeasurementConfidence + = SingleSample + | Range { low: Float, high: Float } + | DistributionSummary { mean: Float, variance: Float } + +type PerformanceReceipt { + provider: ProviderIdentity + work_shape: WorkUnitId + cost: CostAccount + cpu_seconds: Float? + sample_count: Int + measurement_context: ExecutionSurface + confidence: MeasurementConfidence + cache_state_summary: NonEmptyStr? +} + +fn performance_receipts_total_time(receipts: List) -> NanosecondDuration { + nanosecond_duration( + count: fold( + receipts, + init: 0, + f: (acc, receipt) => acc + measure_count(m: receipt.cost.time) + ) + ) +} + +fn cost_account_from_performance_receipts(receipts: List) -> CostAccount { + cost_account_measured_from_time_and_space( + time: performance_receipts_total_time(receipts: receipts), + space: byte_size(count: fold( + receipts, + init: 0, + f: (acc, receipt) => acc + byte_size_count(receipt.cost.space) + )) + ) +} + +data host_v1_interpreter_provider: ProviderIdentity = "host:v1-interpreter" + +data host_v1_interpreter_execution_surface: ExecutionSurface = ExecutionSurface { + os: OperatingSystemSurface { + kernel: Linux, + distro_or_product: none, + version: none, + filesystem_semantics: Posix, + process_semantics: PosixProcess, + }, + isolation: SharedHostHome, + container_runtime: none, + toolchains: [], + mounted_storage: [], + network: [], +} + +fn performance_receipt_host_single_sample( + work_shape: WorkUnitId, + wall_time: NanosecondDuration, + space: ByteSize, +) -> PerformanceReceipt { + PerformanceReceipt { + provider: host_v1_interpreter_provider, + work_shape: work_shape, + cost: cost_account_measured_from_time_and_space(time: wall_time, space: space), + cpu_seconds: none, + sample_count: 1, + measurement_context: host_v1_interpreter_execution_surface, + confidence: SingleSample, + cache_state_summary: none, + } +} + +type PricingSource + = VendorDocument { citation: NonEmptyStr } + | ObservedBill { receipt_id: NonEmptyStr } + | NegotiatedRate { contract_ref: NonEmptyStr } + +type AmortizationScope + = PerLease + | PerCalendarMonth + | PerWorkShape + +type CostReceipt { + provider: ProviderIdentity + cost_class: CostClass + amount: MoneyMicros? + pricing_source: PricingSource + amortization_scope: AmortizationScope? +} + +type ExecutionReceipt { + work: WorkUnit + lease: ComputeLease + output: Outcome> + performance: PerformanceReceipt + cost: CostReceipt + started_at: LogicalTime + finished_at: LogicalTime +} + +data example_host_identity: HostIdentity = "example-host" +data example_shared_host_provider: ProviderIdentity = "example-shared-host" +data example_vm_provider: ProviderIdentity = "example-vm" +data example_availability_open: LogicalTime = "example:always-open" + +data example_linux_os_surface: OperatingSystemSurface = OperatingSystemSurface { + kernel: Linux, + distro_or_product: none, + version: none, + filesystem_semantics: Posix, + process_semantics: PosixProcess, +} + +data example_shared_host_execution_surface: ExecutionSurface = ExecutionSurface { + os: example_linux_os_surface, + isolation: SharedHostHome, + container_runtime: none, + toolchains: [], + mounted_storage: [], + network: [], +} + +data example_vm_execution_surface: ExecutionSurface = ExecutionSurface { + os: example_linux_os_surface, + isolation: Vm, + container_runtime: none, + toolchains: [], + mounted_storage: [], + network: [], +} + +data example_compute_host: ComputeHost = ComputeHost { + identity: example_host_identity, + baseboard: none, + processors: [], + memory: [], + storage: [], + network_interfaces: [], +} + +data example_owned_cost: CostModel = CostModel { + cost_class: OwnedMarginalZero, + rates: ProviderCostModel { placeholder: true }, +} + +data example_shared_host_offer: ComputeOffer = ComputeOffer { + provider: example_shared_host_provider, + supply: ComputeSupplyFacts { + physical: example_compute_host, + execution: example_shared_host_execution_surface, + cost: Present { value: example_owned_cost }, + observed_performance: [], + }, + available_window: AvailabilityWindow { open: example_availability_open, close: none }, + cost_quote: none, + constraints: [], +} + +data example_vm_offer: ComputeOffer = ComputeOffer { + provider: example_vm_provider, + supply: ComputeSupplyFacts { + physical: example_compute_host, + execution: example_vm_execution_surface, + cost: Present { value: example_owned_cost }, + observed_performance: [], + }, + available_window: AvailabilityWindow { open: example_availability_open, close: none }, + cost_quote: none, + constraints: [], +} + +data vm_isolation_work_demand: WorkDemand = WorkDemand { + resources: ResourceEnvelope { + cpu: none, + gpu: none, + memory: none, + storage: none, + network: none, + }, + os: none, + isolation: IsolationRequirement { boundary: Vm }, + toolchains: [], + parallelism: SingleWorkItem, + data_locality: [], + effects: [], + input_envelope: EnvelopeUnknown, +} + +fn witness_shared_host_home_rejects_vm() -> Bool { + isolation_satisfies(supply: SharedHostHome, demand: Vm) == false +} + +fn witness_shared_host_home_admits_shared_host_home() -> Bool { + isolation_satisfies(supply: SharedHostHome, demand: SharedHostHome) +} + +fn witness_vm_admits_shared_host_home() -> Bool { + isolation_satisfies(supply: Vm, demand: SharedHostHome) +} + +fn witness_shared_host_offer_rejects_vm_demand() -> Bool { + match satisfies(offer: example_shared_host_offer, demand: vm_isolation_work_demand) { + Rejected { reason: _ } => true + Eligible { witness: _ } => false + } +} + +fn witness_vm_offer_admits_vm_demand() -> Bool { + match satisfies(offer: example_vm_offer, demand: vm_isolation_work_demand) { + Eligible { witness: _ } => true + Rejected { reason: _ } => false + } +} + +data example_resource_cpu_catalog: CpuModelCatalogRow = CpuModelCatalogRow { + model_number: "GENERIC-ARM-64C", + oem_listing_id: none, + vendor: ampere, + architecture: Aarch64, + cores: 64, + threads: hardware_thread_count(64), + nominal_sustained_per_thread_hz: hertz(3000000000), + socket: CpuSocket { package: LandGridArray, contact_count: 4926 }, + tdp_watts: watt(250), +} + +data example_resource_cpu: CpuFacts = CpuFacts { + catalog: example_resource_cpu_catalog, + deployment: CpuDeploymentFacts { sustained_per_thread_hz: hertz(3000000000) }, +} + +data example_resource_host: ComputeHost = ComputeHost { + identity: "example-resource-host", + baseboard: none, + processors: [CpuProcessor { cpu: example_resource_cpu }], + memory: [MemoryDevice { capacity: byte_size(68719476736), memory_kind: Dram, bandwidth: none }], + storage: [], + network_interfaces: [], +} + +data example_resource_offer: ComputeOffer = ComputeOffer { + provider: "example-resource-offer", + supply: ComputeSupplyFacts { + physical: example_resource_host, + execution: example_shared_host_execution_surface, + cost: Present { value: example_owned_cost }, + observed_performance: [], + }, + available_window: AvailabilityWindow { open: example_availability_open, close: none }, + cost_quote: none, + constraints: [], +} + +data example_thread_work_demand: WorkDemand = WorkDemand { + resources: ResourceEnvelope { + cpu: Present { + value: CpuRequirement { + min_threads: hardware_thread_count(128), + architecture: Present { value: Aarch64 }, + }, + }, + gpu: none, + memory: none, + storage: none, + network: none, + }, + os: none, + isolation: IsolationRequirement { boundary: SharedHostHome }, + toolchains: [], + parallelism: SingleWorkItem, + data_locality: [], + effects: [], + input_envelope: EnvelopeUnknown, +} + +data example_memory_work_demand: WorkDemand = WorkDemand { + resources: ResourceEnvelope { + cpu: none, + gpu: none, + memory: Present { value: MemoryRequirement { min_bytes: byte_size(137438953472) } }, + storage: none, + network: none, + }, + os: none, + isolation: IsolationRequirement { boundary: SharedHostHome }, + toolchains: [], + parallelism: SingleWorkItem, + data_locality: [], + effects: [], + input_envelope: EnvelopeUnknown, +} + +data example_feasible_resource_work_demand: WorkDemand = WorkDemand { + resources: ResourceEnvelope { + cpu: Present { + value: CpuRequirement { + min_threads: hardware_thread_count(2), + architecture: Present { value: Aarch64 }, + }, + }, + gpu: none, + memory: Present { value: MemoryRequirement { min_bytes: byte_size(1073741824) } }, + storage: none, + network: none, + }, + os: none, + isolation: IsolationRequirement { boundary: SharedHostHome }, + toolchains: [], + parallelism: SingleWorkItem, + data_locality: [], + effects: [], + input_envelope: EnvelopeUnknown, +} + +fn witness_resource_offer_rejects_insufficient_threads() -> Bool { + match satisfies(offer: example_resource_offer, demand: example_thread_work_demand) { + Rejected { reason: r } => r.dimension == DemandCpu + Eligible { witness: _ } => false + } +} + +fn witness_resource_offer_rejects_insufficient_memory() -> Bool { + match satisfies(offer: example_resource_offer, demand: example_memory_work_demand) { + Rejected { reason: r } => r.dimension == DemandMemory + Eligible { witness: _ } => false + } +} + +fn witness_resource_offer_admits_feasible_demand() -> Bool { + match satisfies(offer: example_resource_offer, demand: example_feasible_resource_work_demand) { + Eligible { witness: w } => + w.satisfied.length() == 3 + && w.satisfied.first() == DemandIsolation + && w.satisfied.skip(n: 1).first() == DemandCpu + && w.satisfied.skip(n: 2).first() == DemandMemory + Rejected { reason: _ } => false + } +} + +fn witness_run_demand_envelope_fits_budget() -> Bool { + let envelope = parallel_run_demand_envelope_from_budget( + shard_count: 7, + hardware_threads: hardware_thread_count(128), + memory_budget: byte_size(67108864000), + per_shard_peak: byte_size(2100028416), + conservative_fallback_width: 4 + ) + demand_envelope_thread_count(envelope: envelope) == 7 + && demand_envelope_fits_budget(envelope: envelope, budget: byte_size(67108864000)) +} + +fn witness_high_demand_shrinks_envelope_and_fits() -> Bool { + let envelope = parallel_run_demand_envelope_from_budget( + shard_count: 7, + hardware_threads: hardware_thread_count(128), + memory_budget: byte_size(67108864000), + per_shard_peak: byte_size(20000000000), + conservative_fallback_width: 4 + ) + demand_envelope_thread_count(envelope: envelope) == 2 + && demand_envelope_fits_budget(envelope: envelope, budget: byte_size(67108864000)) +} + +fn witness_fail_open_memory_blind_demand_exceeds_budget() -> Bool { + let blind_width = bounded_host_spawn_width( + shard_count: 7, + hardware_threads: hardware_thread_count(128) + ) + let blind_envelope = parallel_run_demand_envelope( + width: blind_width, + per_shard_peak: byte_size(20000000000) + ) + demand_envelope_fits_budget(envelope: blind_envelope, budget: byte_size(67108864000)) == false +} + +fn witness_unreadable_budget_uses_conservative_demand() -> Bool { + let envelope = parallel_run_demand_envelope_from_budget( + shard_count: 7, + hardware_threads: hardware_thread_count(128), + memory_budget: byte_size(0), + per_shard_peak: byte_size(2100028416), + conservative_fallback_width: 4 + ) + demand_envelope_thread_count(envelope: envelope) == 4 +} + +data execution_receipt_digest_schema: NonEmptyStr = "execution-receipt-canonical-v1" + +fn isolation_boundary_digest(boundary: IsolationBoundary) -> ContentHash { + content_hash_atom(value: isolation_boundary_label(boundary: boundary)) +} + +fn cpu_architecture_digest(arch: Architecture) -> ContentHash { + match arch { + X86_64 => content_hash_atom(value: "arch:X86_64") + X86 => content_hash_atom(value: "arch:X86") + Aarch64 => content_hash_atom(value: "arch:Aarch64") + Arm => content_hash_atom(value: "arch:Arm") + Armv7 => content_hash_atom(value: "arch:Armv7") + Mips => content_hash_atom(value: "arch:Mips") + Mipsel => content_hash_atom(value: "arch:Mipsel") + Mips64 => content_hash_atom(value: "arch:Mips64") + Mips64el => content_hash_atom(value: "arch:Mips64el") + Riscv64 => content_hash_atom(value: "arch:Riscv64") + Wasm32 => content_hash_atom(value: "arch:Wasm32") + } +} + +fn optional_cpu_requirement_digest(req: CpuRequirement?) -> ContentHash { + match req { + Absent => content_hash_atom(value: "cpu:none") + Present { value: r } => + content_hash_combine( + content_hash_atom(value: "cpu"), + content_hash_combine( + content_hash_atom(value: to_string(hardware_thread_count_value(r.min_threads))), + match r.architecture { + Absent => content_hash_atom(value: "arch:any") + Present { value: arch } => cpu_architecture_digest(arch: arch) + }, + ), + ) + } +} + +fn optional_memory_requirement_digest(req: MemoryRequirement?) -> ContentHash { + match req { + Absent => content_hash_atom(value: "memory:none") + Present { value: r } => + content_hash_combine( + content_hash_atom(value: "memory"), + content_hash_atom(value: to_string(byte_size_count(r.min_bytes))), + ) } } -fn witness_hard_requirement_unmet_is_unmet() -> Bool { - let big_need = ComputeNeed { fabric: example_fabric, program: example_large_program } - match connect(need: big_need, opportunities: [example_small_opportunity]) { - Unmet { reason: _ } => true - Bound { opportunity: _ } => false - Pending => false +fn resource_envelope_digest(envelope: ResourceEnvelope) -> ContentHash { + content_hash_combine( + optional_cpu_requirement_digest(req: envelope.cpu), + content_hash_combine( + optional_memory_requirement_digest(req: envelope.memory), + content_hash_atom(value: "resource-envelope-v1"), + ), + ) +} + +fn parallelism_shape_digest(shape: ParallelismShape) -> ContentHash { + match shape { + SingleWorkItem => content_hash_atom(value: "parallelism:SingleWorkItem") + IndependentShards { shard_count: n } => + content_hash_combine( + content_hash_atom(value: "parallelism:IndependentShards"), + content_hash_atom(value: to_string(n)), + ) + DependencyGraphParallel { graph: graph } => + content_hash_combine( + content_hash_atom(value: "parallelism:DependencyGraphParallel"), + content_hash_atom(value: "parallelism-payload-pending:WorkGraph"), + ) + PartitionedReduce { partitioner: partitioner, map: map, reduce: reduce, laws: laws } => + content_hash_combine( + content_hash_atom(value: "parallelism:PartitionedReduce"), + content_hash_combine( + content_hash_atom(value: "parallelism-payload-pending:Partitioner"), + content_hash_combine( + content_hash_atom(value: "parallelism-payload-pending:WorkAction-map"), + content_hash_combine( + content_hash_atom(value: "parallelism-payload-pending:Reducer"), + content_hash_atom(value: "parallelism-payload-pending:ReducerLaws"), + ), + ), + ), + ) + } +} + +fn work_demand_content_digest(demand: WorkDemand) -> ContentHash { + content_hash_combine( + resource_envelope_digest(envelope: demand.resources), + content_hash_combine( + isolation_boundary_digest(boundary: demand.isolation.boundary), + content_hash_combine( + parallelism_shape_digest(shape: demand.parallelism), + content_hash_atom(value: "work-demand-v1"), + ), + ), + ) +} + +fn execution_receipt_subject_digest( + work_id: WorkUnitId, + demand: WorkDemand, +) -> ExecutionReceiptDigest { + content_hash_tagged( + tag: execution_receipt_digest_schema, + payload: content_hash_combine( + content_hash_atom(value: work_id), + work_demand_content_digest(demand: demand), + ), + ) +} + +fn execution_receipt_digest(receipt: ExecutionReceipt) -> ExecutionReceiptDigest { + execution_receipt_subject_digest( + work_id: receipt.work.id, + demand: receipt.work.requirements, + ) +} + +fn witness_execution_receipt_digest_not_raw_work_id( + work_id: WorkUnitId, + demand: WorkDemand, +) -> Bool { + execution_receipt_subject_digest(work_id: work_id, demand: demand) != work_id +} + +fn witness_execution_receipt_digest_differs_by_demand( + work_id: WorkUnitId, + left: WorkDemand, + right: WorkDemand, +) -> Bool { + execution_receipt_subject_digest(work_id: work_id, demand: left) != execution_receipt_subject_digest(work_id: work_id, demand: right) +} + +data gunbc_ci_corpus_envelope: InputEnvelope = BoundedInput { + bounds: [ + InputBound { + axis: WitnessCount, + max: Measure { count: 2000 } + }, + InputBound { + axis: SourceNodeCount, + max: Measure { count: 500000 } + }, + InputBound { + axis: CorpusNodeCount, + max: Measure { count: 5000000 } + }, + ] +} + +data gunbc_ci_corpus_envelope_ceiling_scaffold: Disposition = Scaffold { + dissolves_to: SingleAuthority, + bind: DeclarationRef { + module_path: "gunbc.plans.input_envelope_roadmap", + decl_name: "input_envelope_roadmap_plan", + field: WholeDeclaration } } diff --git a/dsl/test/claim/compute_fabric_resource_witness_test.dag b/dsl/test/claim/compute_fabric_resource_witness_test.dag new file mode 100644 index 00000000000..ca14963d86e --- /dev/null +++ b/dsl/test/claim/compute_fabric_resource_witness_test.dag @@ -0,0 +1,24 @@ +module test.claim.compute_fabric_resource_witness + +import product.compute_fabric { + witness_resource_offer_rejects_insufficient_threads, + witness_resource_offer_rejects_insufficient_memory, + witness_resource_offer_admits_feasible_demand, + witness_run_demand_envelope_fits_budget, + witness_high_demand_shrinks_envelope_and_fits, + witness_fail_open_memory_blind_demand_exceeds_budget, + witness_unreadable_budget_uses_conservative_demand, +} + +test fn compute_fabric_resource_feasibility_holds() -> Bool { + witness_resource_offer_rejects_insufficient_threads() + && witness_resource_offer_rejects_insufficient_memory() + && witness_resource_offer_admits_feasible_demand() +} + +test fn ci_floor_run_demand_envelope_model_holds() -> Bool { + witness_run_demand_envelope_fits_budget() + && witness_high_demand_shrinks_envelope_and_fits() + && witness_fail_open_memory_blind_demand_exceeds_budget() + && witness_unreadable_budget_uses_conservative_demand() +} diff --git a/dsl/test/claim/compute_fabric_witness_test.dag b/dsl/test/claim/compute_fabric_witness_test.dag deleted file mode 100644 index 21c1a201259..00000000000 --- a/dsl/test/claim/compute_fabric_witness_test.dag +++ /dev/null @@ -1,20 +0,0 @@ -module test.claim.compute_fabric_witness - -import product.compute_fabric { - witness_need_is_only_fabric_plus_program, - witness_shape_is_derived_from_program, - witness_fabric_binds_need_to_opportunity, - witness_exec_and_ci_are_just_programs, - witness_hard_requirement_unmet_is_unmet, -} - -test fn compute_fabric_broker_shape_and_need_holds() -> Bool { - witness_need_is_only_fabric_plus_program() - && witness_shape_is_derived_from_program() -} - -test fn compute_fabric_broker_connect_holds() -> Bool { - witness_fabric_binds_need_to_opportunity() - && witness_exec_and_ci_are_just_programs() - && witness_hard_requirement_unmet_is_unmet() -} diff --git a/dsl/test/claim/probe_selector_compute_fabric_import_repro_test.dag b/dsl/test/claim/probe_selector_compute_fabric_import_repro_test.dag index 57f2f1a412f..5c32375d951 100644 --- a/dsl/test/claim/probe_selector_compute_fabric_import_repro_test.dag +++ b/dsl/test/claim/probe_selector_compute_fabric_import_repro_test.dag @@ -1,7 +1,7 @@ module test.claim.probe_selector_compute_fabric_import_repro -import product.compute_fabric { example_fabric } +import product.compute_fabric { example_shared_host_offer } test fn probe_selector_compute_fabric_import_repro_holds() -> Bool { - example_fabric == example_fabric + example_shared_host_offer.provider == example_shared_host_offer.provider } diff --git a/dsl/test/claim/roadmap_authority_test.dag b/dsl/test/claim/roadmap_authority_test.dag index 78b1b0f56b6..36e810f6011 100644 --- a/dsl/test/claim/roadmap_authority_test.dag +++ b/dsl/test/claim/roadmap_authority_test.dag @@ -63,6 +63,11 @@ fn witness_c4_selfhost_no_overclaim() -> Bool { && string_contains(s: o, pattern: "**TERMINAL: `src/v1` deleted**") } +fn witness_c4_cargo_green_landed() -> Bool { + string_contains(s: out(), + pattern: "- [x] **emitted crate `cargo build`s green (Route-A last mile) ✓ LANDED (#5777/#5873)**") +} + fn witness_c5_rust_gate_not_greenlit() -> Bool { string_contains(s: out(), pattern: "- [ ] **rust-gate coverage** (shared §1)") } @@ -99,6 +104,7 @@ test fn roadmap_authority_witnesses() -> Bool { && witness_c2_roundtrip_law() && witness_c3_inversion_subrows() && witness_c4_selfhost_no_overclaim() + && witness_c4_cargo_green_landed() && witness_c5_rust_gate_not_greenlit() && witness_a_prose_verbatim() && witness_b_ordered_interleaving() diff --git a/src/v1/04_infer.dag b/src/v1/04_infer.dag index 7384e8242bb..e1657e25ca5 100644 --- a/src/v1/04_infer.dag +++ b/src/v1/04_infer.dag @@ -280,7 +280,7 @@ type CycleDetectState { } type InferScopeComponents { - imported_sigs: Map + func_sigs: Map svc_registry: Map> svc_locals: Map } @@ -295,20 +295,32 @@ type LocalContributionState { } fn merge_scope_from_imports(remaining: List, parent_index: Map, env: TypeEnv, - imported_sigs: Map, + func_sigs: Map, svc_registry: Map>, svc_locals: Map) -> InferScopeComponents { match remaining |> first { Absent => InferScopeComponents { - imported_sigs: imported_sigs, svc_registry: svc_registry, + func_sigs: func_sigs, svc_registry: svc_registry, svc_locals: svc_locals } Present { value: imp } => match map_get(parent_index, imp.module_path) { Present { value: typed_parent } => - let next_imported_sigs = map_merge(imported_sigs, typed_parent.func_env.signatures) + let next_func_sigs = fold(typed_parent.func_env.signatures |> map_values, init: func_sigs, f: (acc, rsig) => + let sig_key = rsig.name + let sig_params = rsig.params + let sig_rt = rsig.inferred + let sig_async = rsig.is_async + map_insert(acc, sig_key, DeclaredFuncSig { + name: sig_key, params: sig_params, + inferred: Present { value: sig_rt }, + is_async: sig_async, + output_provenance: rsig.output_provenance, + variant_provenance: rsig.variant_provenance + }) + ) let parent_result = fold(typed_parent.items, init: InferScopeComponents { - imported_sigs: next_imported_sigs, svc_registry: svc_registry, + func_sigs: next_func_sigs, svc_registry: svc_registry, svc_locals: svc_locals }, f: (acc, titem) => if titem.transport != none && titem.children |> count > 0 { @@ -317,14 +329,14 @@ fn merge_scope_from_imports(remaining: List, parent_index: Map count == 0) && titem.transport == none && titem.connective == NoConnective && titem.inferred != none { InferScopeComponents { - imported_sigs: acc.imported_sigs, + func_sigs: acc.func_sigs, svc_registry: acc.svc_registry, svc_locals: map_insert(acc.svc_locals, authored_name_at(source_indices: env.source_indices, node: titem), TypeBinding { name: authored_name_at(source_indices: env.source_indices, node: titem), resolved: resolved_type(n: titem), provenance: SubValueUnknown }) } @@ -334,11 +346,11 @@ fn merge_scope_from_imports(remaining: List, parent_index: Map skip(1), parent_index: parent_index, env: env, - imported_sigs: parent_result.imported_sigs, svc_registry: parent_result.svc_registry, + func_sigs: parent_result.func_sigs, svc_registry: parent_result.svc_registry, svc_locals: parent_result.svc_locals) Absent => merge_scope_from_imports(remaining: remaining |> skip(1), parent_index: parent_index, env: env, - imported_sigs: imported_sigs, svc_registry: svc_registry, + func_sigs: func_sigs, svc_registry: svc_registry, svc_locals: svc_locals) } } @@ -5824,16 +5836,16 @@ fn build_module_context(contributions: List, remaining: resolved_imports, parent_index: parent_index, env: env, - imported_sigs: empty_map(), + func_sigs: empty_map(), svc_registry: local.svc_registry, svc_locals: local.svc_locals ) + let all_declared_sigs = map_merge(merged_scope.func_sigs, local.func_sigs) let resolve_result = resolve_func_sigs( - declared_sigs: local.func_sigs, + declared_sigs: all_declared_sigs, items: local.resolved_items, module_name: module_name, - source_indices: env.source_indices, - parent_sigs: merged_scope.imported_sigs + source_indices: env.source_indices ) let all_locals = fold(merged_scope.svc_locals |> map_values, init: env_variant_locals, f: (acc, binding) => map_insert(acc, binding.name, binding) diff --git a/src/v1/04_sigs.dag b/src/v1/04_sigs.dag index 94942ee0ffb..e92b595c069 100644 --- a/src/v1/04_sigs.dag +++ b/src/v1/04_sigs.dag @@ -81,6 +81,18 @@ fn build_name_set(names: List) -> Map { fold(names, init: empty_map(), f: (acc, name) => map_insert(acc, name, true)) } +fn collect_parent_resolved_sigs(declared_sigs: Map, local_func_set: Map) -> Map { + fold(map_values(declared_sigs), init: empty_map(), f: (acc, dsig) => + if emit_map_has(m: local_func_set, key: dsig.name) { + acc + } else if dsig.inferred != none { + map_insert(acc, dsig.name, declared_to_resolved(dsig: dsig)) + } else { + acc + } + ) +} + fn declared_to_resolved(dsig: DeclaredFuncSig) -> ResolvedFuncSig { ResolvedFuncSig { name: dsig.name, params: dsig.params, inferred: dsig.inferred.value, is_async: dsig.is_async, output_provenance: dsig.output_provenance, variant_provenance: dsig.variant_provenance } } @@ -177,21 +189,17 @@ fn resolve_func_sigs( declared_sigs: Map, items: List, module_name: String, - source_indices: Map, - parent_sigs: Map + source_indices: Map ) -> ResolveFuncSigsResult { let local_func_names = items |> filter(item => item.params |> count > 0 && item.body != none) |> map(item => authored_name_at(source_indices: source_indices, node: item)) let local_func_set = build_name_set(names: local_func_names) let call_edges = collect_func_call_edges(items: items, local_func_set: local_func_set, source_indices: source_indices) - let parent_sigs_filtered = fold(map_values(parent_sigs), init: empty_map(), f: (acc, rsig) => - if emit_map_has(m: local_func_set, key: rsig.name) { acc } - else { map_insert(acc, rsig.name, rsig) } - ) + let parent_resolved = collect_parent_resolved_sigs(declared_sigs: declared_sigs, local_func_set: local_func_set) topo_resolve_loop( remaining: local_func_names, - resolved: parent_sigs_filtered, + resolved: parent_resolved, declared_sigs: declared_sigs, call_edges: call_edges, local_func_set: local_func_set, diff --git a/src/v1/stage0/Cargo.toml b/src/v1/stage0/Cargo.toml index eaf2e38bd23..0dfef8fe442 100644 --- a/src/v1/stage0/Cargo.toml +++ b/src/v1/stage0/Cargo.toml @@ -92,10 +92,3 @@ path = "src/bin/v1_src_dag_parse.rs" [[bin]] name = "wiring_liveness_whole_tree" path = "src/bin/wiring_liveness_whole_tree.rs" - -# Measurement binary: walks MultiEntryIndex caches after a resolve and reports -# shared-vs-distinct Rc<> pointer stats for typed_module_cache, parse_cache. -# Used to diagnose ~4 GiB retention root cause before cutting a fix. -[[bin]] -name = "cache_walk" -path = "src/bin/cache_walk.rs" diff --git a/src/v1/stage0/src/bin/cache_walk.rs b/src/v1/stage0/src/bin/cache_walk.rs deleted file mode 100644 index 29bb541e665..00000000000 --- a/src/v1/stage0/src/bin/cache_walk.rs +++ /dev/null @@ -1,72 +0,0 @@ -// SCAFFOLD: diagnostic bin for typed_module_cache retention measurement; delete once func_env.sigs -// Rc-sharing dedup lands. Dissolution trigger = func_env.sig sharing > 1.0× in corpus. -#![allow(clippy::disallowed_macros)] - -use std::process::ExitCode; - -use v1_compiler::cli_run::{build_multi_entry_index, resolve_entry_with_index}; - -fn usage() -> ! { - eprintln!("cache_walk: measure MultiEntryIndex cache structure after resolving entries"); - eprintln!( - "usage: cache_walk --source-root [--source-root ...] --entry [--entry ...]" - ); - std::process::exit(1); -} - -fn main() -> ExitCode { - let args: Vec = std::env::args().skip(1).collect(); - let mut source_roots: Vec = Vec::new(); - let mut entries: Vec = Vec::new(); - let mut i = 0; - while i < args.len() { - match args[i].as_str() { - "--source-root" => { - i += 1; - if i >= args.len() { - usage(); - } - source_roots.push(args[i].clone()); - } - "--entry" => { - i += 1; - if i >= args.len() { - usage(); - } - entries.push(args[i].clone()); - } - _ => { - eprintln!("unknown arg: {}", args[i]); - usage(); - } - } - i += 1; - } - if source_roots.is_empty() || entries.is_empty() { - usage(); - } - - eprintln!("cache_walk: building index for {:?}", source_roots); - let index = build_multi_entry_index(&source_roots); - - for entry in &entries { - eprintln!("cache_walk: resolving entry {entry}"); - match resolve_entry_with_index(&index, entry) { - Err(e) => { - eprintln!("cache_walk: resolve failed for {entry}: {e}"); - return ExitCode::FAILURE; - } - Ok(_) => { - eprintln!( - "cache_walk: resolve OK ({} modules cached so far)", - index.typed_module_count() - ); - } - } - } - - let report = index.measure_caches(); - println!("{}", report.format()); - - ExitCode::SUCCESS -} diff --git a/src/v1/stage0/src/cli_run.rs b/src/v1/stage0/src/cli_run.rs index 93341391cde..65ee1878cc1 100644 --- a/src/v1/stage0/src/cli_run.rs +++ b/src/v1/stage0/src/cli_run.rs @@ -324,182 +324,6 @@ pub struct MultiEntryIndex { typed_module_cache: RefCell>>, } -// SCAFFOLD: delete once func_env.sigs Rc-sharing dedup lands; dissolution trigger = func_env.sig sharing > 1.0× in corpus. -pub struct CacheWalkReport { - pub typed_module_count: usize, - pub parse_cache_entry_count: usize, - // typed_module_cache: TypeEnv.bindings - pub total_bindings: usize, - pub distinct_binding_ptrs: usize, - // typed_module_cache: TypeEnv.source_indices - pub total_source_index_entries: usize, - pub distinct_source_index_ptrs: usize, - // typed_module_cache: TypeEnv.inductive_fields (map entry count) - pub total_inductive_field_entries: usize, - pub distinct_inductive_field_vec_ptrs: usize, - // typed_module_cache: TypeEnv.intern_table - pub distinct_intern_table_ptrs: usize, - // typed_module_cache: func_env.signatures - pub total_func_sigs: usize, - pub distinct_func_sig_ptrs: usize, - // typed_module_cache: typed items (Rc in items vec) - pub total_typed_items: usize, - pub distinct_typed_item_node_ptrs: usize, - // parse_cache: parsed module nodes (Rc) - pub total_parse_module_nodes: usize, - pub distinct_parse_module_node_ptrs: usize, -} - -impl CacheWalkReport { - pub fn format(&self) -> String { - let sharing = |total: usize, distinct: usize| -> String { - if distinct == 0 { - return "n/a".to_string(); - } - format!("{:.1}×", total as f64 / distinct as f64) - }; - format!( - "=== MultiEntryIndex cache walk ===\n\ - typed_module_cache: {} modules\n\ - parse_cache: {} files\n\ - \n\ - TypeEnv.bindings:\n\ - total entries: {}\n\ - distinct Rc<>: {} (sharing {}, ideal 1×)\n\ - TypeEnv.source_indices entries:\n\ - total: {}\n\ - distinct Rc: {} (sharing {})\n\ - TypeEnv.inductive_fields entries:\n\ - total: {}\n\ - distinct Rc>: {} (sharing {})\n\ - TypeEnv.intern_table:\n\ - distinct Rc<>: {} (sharing {})\n\ - func_env.signatures:\n\ - total: {}\n\ - distinct Rc<>: {} (sharing {})\n\ - typed items (Rc in items):\n\ - total: {}\n\ - distinct Rc<>: {} (sharing {})\n\ - parse module nodes:\n\ - total: {}\n\ - distinct Rc<>: {} (sharing {})\n", - self.typed_module_count, - self.parse_cache_entry_count, - self.total_bindings, - self.distinct_binding_ptrs, - sharing(self.total_bindings, self.distinct_binding_ptrs), - self.total_source_index_entries, - self.distinct_source_index_ptrs, - sharing( - self.total_source_index_entries, - self.distinct_source_index_ptrs - ), - self.total_inductive_field_entries, - self.distinct_inductive_field_vec_ptrs, - sharing( - self.total_inductive_field_entries, - self.distinct_inductive_field_vec_ptrs - ), - self.distinct_intern_table_ptrs, - sharing(self.typed_module_count, self.distinct_intern_table_ptrs), - self.total_func_sigs, - self.distinct_func_sig_ptrs, - sharing(self.total_func_sigs, self.distinct_func_sig_ptrs), - self.total_typed_items, - self.distinct_typed_item_node_ptrs, - sharing(self.total_typed_items, self.distinct_typed_item_node_ptrs), - self.total_parse_module_nodes, - self.distinct_parse_module_node_ptrs, - sharing( - self.total_parse_module_nodes, - self.distinct_parse_module_node_ptrs - ), - ) - } -} - -impl MultiEntryIndex { - pub fn typed_module_count(&self) -> usize { - self.typed_module_cache.borrow().len() - } - - pub fn measure_caches(&self) -> CacheWalkReport { - use crate::v1_compiler_infer_env::TypeEnv; - use std::collections::HashSet; - - let typed_cache = self.typed_module_cache.borrow(); - let parse_cache_ref = self.parse_cache.borrow(); - - let mut total_bindings = 0usize; - let mut binding_ptrs: HashSet<*const crate::v1_compiler_infer_env::TypeBinding> = - HashSet::new(); - let mut total_si_entries = 0usize; - let mut si_ptrs: HashSet<*const NewlineIndex> = HashSet::new(); - let mut total_if_entries = 0usize; - let mut if_vec_ptrs: HashSet = HashSet::new(); - let mut intern_ptrs: HashSet<*const InternTable> = HashSet::new(); - let mut total_func_sigs = 0usize; - let mut sig_ptrs: HashSet<*const crate::v1_compiler_infer_sigs::ResolvedFuncSig> = - HashSet::new(); - let mut total_typed_items = 0usize; - let mut typed_item_ptrs: HashSet<*const Node> = HashSet::new(); - - for tc_result in typed_cache.values() { - let tm = &tc_result.typed; - let env = &tm.type_env; - - for binding in env.bindings.values() { - total_bindings += 1; - binding_ptrs.insert(Rc::as_ptr(binding)); - } - for si in env.source_indices.values() { - total_si_entries += 1; - si_ptrs.insert(Rc::as_ptr(si)); - } - for iv in env.inductive_fields.values() { - total_if_entries += 1; - if_vec_ptrs.insert(Rc::as_ptr(iv) as usize); - } - intern_ptrs.insert(Rc::as_ptr(&env.intern_table)); - for sig in tm.func_env.signatures.values() { - total_func_sigs += 1; - sig_ptrs.insert(Rc::as_ptr(sig)); - } - for item in tm.items.iter() { - total_typed_items += 1; - typed_item_ptrs.insert(Rc::as_ptr(item)); - } - } - - let mut total_parse_nodes = 0usize; - let mut parse_node_ptrs: HashSet<*const Node> = HashSet::new(); - for (parse_result, _) in parse_cache_ref.values() { - if let Some(module) = &parse_result.module { - total_parse_nodes += 1; - parse_node_ptrs.insert(Rc::as_ptr(module)); - } - } - - CacheWalkReport { - typed_module_count: typed_cache.len(), - parse_cache_entry_count: parse_cache_ref.len(), - total_bindings, - distinct_binding_ptrs: binding_ptrs.len(), - total_source_index_entries: total_si_entries, - distinct_source_index_ptrs: si_ptrs.len(), - total_inductive_field_entries: total_if_entries, - distinct_inductive_field_vec_ptrs: if_vec_ptrs.len(), - distinct_intern_table_ptrs: intern_ptrs.len(), - total_func_sigs, - distinct_func_sig_ptrs: sig_ptrs.len(), - total_typed_items, - distinct_typed_item_node_ptrs: typed_item_ptrs.len(), - total_parse_module_nodes: total_parse_nodes, - distinct_parse_module_node_ptrs: parse_node_ptrs.len(), - } - } -} - fn seed_kernel_intern_names(table: Rc) -> Rc { let mut t = table; for name in v1_rt::map_keys(&kernel_type_set()).iter().cloned() { diff --git a/src/v1/stage0/src/v1_compiler_infer.rs b/src/v1/stage0/src/v1_compiler_infer.rs index d42495eb924..0435ff8be85 100644 --- a/src/v1/stage0/src/v1_compiler_infer.rs +++ b/src/v1/stage0/src/v1_compiler_infer.rs @@ -339,7 +339,7 @@ pub struct CycleDetectState { #[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)] pub struct InferScopeComponents { - pub imported_sigs: Rc>>, + pub func_sigs: Rc>>, pub svc_registry: Rc>>>>, pub svc_locals: Rc>>, } @@ -358,7 +358,7 @@ pub fn merge_scope_from_imports( mut remaining: Rc>>, mut parent_index: Rc>>, mut env: Rc, - mut imported_sigs: Rc>>, + mut func_sigs: Rc>>, mut svc_registry: Rc>>>>, mut svc_locals: Rc>>, ) -> Rc { @@ -366,20 +366,43 @@ pub fn merge_scope_from_imports( match remaining.clone().first().cloned() { None => { break Rc::new(InferScopeComponents { - imported_sigs: imported_sigs, + func_sigs: func_sigs, svc_registry: svc_registry, svc_locals: svc_locals, }); } Some(imp) => match v1_rt::map_get(&parent_index, imp.module_path.clone()) { Some(typed_parent) => { - let next_imported_sigs = v1_rt::rc_map_merge( - imported_sigs, - typed_parent.func_env.clone().signatures.clone(), + let next_func_sigs = Rc::new(v1_rt::map_values( + &typed_parent.func_env.clone().signatures.clone(), + )) + .iter() + .cloned() + .fold( + func_sigs, + |acc: Rc>>, + rsig: Rc| { + let sig_key = rsig.name.clone(); + let sig_params = rsig.params.clone(); + let sig_rt = rsig.inferred.clone(); + let sig_async = rsig.is_async.clone(); + v1_rt::rc_map_insert( + acc, + sig_key.clone(), + Rc::new(DeclaredFuncSig { + name: sig_key.clone(), + params: sig_params.clone(), + inferred: Some(sig_rt.clone()), + is_async: sig_async.clone(), + output_provenance: rsig.output_provenance.clone(), + variant_provenance: rsig.variant_provenance.clone(), + }), + ) + }, ); let parent_result = typed_parent.items.clone().iter().cloned().fold( Rc::new(InferScopeComponents { - imported_sigs: next_imported_sigs, + func_sigs: next_func_sigs, svc_registry: svc_registry, svc_locals: svc_locals, }), @@ -412,7 +435,7 @@ pub fn merge_scope_from_imports( env.source_indices.clone(), ); Rc::new(InferScopeComponents { - imported_sigs: acc.imported_sigs.clone(), + func_sigs: acc.func_sigs.clone(), svc_registry: v1_rt::rc_map_insert( acc.svc_registry.clone(), authored_name_at( @@ -436,7 +459,7 @@ pub fn merge_scope_from_imports( && (titem.inferred.clone() != None)) { Rc::new(InferScopeComponents { - imported_sigs: acc.imported_sigs.clone(), + func_sigs: acc.func_sigs.clone(), svc_registry: acc.svc_registry.clone(), svc_locals: v1_rt::rc_map_insert( acc.svc_locals.clone(), @@ -470,11 +493,11 @@ pub fn merge_scope_from_imports( .skip(1 as usize) .collect::>(), ); - let __tco_1 = parent_result.imported_sigs.clone(); + let __tco_1 = parent_result.func_sigs.clone(); let __tco_2 = parent_result.svc_registry.clone(); let __tco_3 = parent_result.svc_locals.clone(); remaining = __tco_0; - imported_sigs = __tco_1; + func_sigs = __tco_1; svc_registry = __tco_2; svc_locals = __tco_3; continue; @@ -13068,16 +13091,17 @@ if (curr_is_imported.clone() && prev_is_imported.clone()) { resolved_imports.clone(), parent_index, env.clone(), - v1_rt::rc_empty_map::>(), + v1_rt::rc_empty_map::>(), local.svc_registry.clone(), local.svc_locals.clone(), ); + let all_declared_sigs = + v1_rt::rc_map_merge(merged_scope.func_sigs.clone(), local.func_sigs.clone()); let resolve_result = resolve_func_sigs( - local.func_sigs.clone(), + all_declared_sigs, local.resolved_items.clone(), module_name.clone(), env.source_indices.clone(), - merged_scope.imported_sigs.clone(), ); let all_locals = Rc::new(v1_rt::map_values(&merged_scope.svc_locals.clone())) .iter() diff --git a/src/v1/stage0/src/v1_compiler_infer_sigs.rs b/src/v1/stage0/src/v1_compiler_infer_sigs.rs index 9d3fec85608..50aaab3357e 100644 --- a/src/v1/stage0/src/v1_compiler_infer_sigs.rs +++ b/src/v1/stage0/src/v1_compiler_infer_sigs.rs @@ -185,6 +185,33 @@ pub fn build_name_set(names: Rc>) -> Rc> { ) } +pub fn collect_parent_resolved_sigs( + declared_sigs: Rc>>, + local_func_set: Rc>, +) -> Rc>> { + Rc::new(v1_rt::map_values(&declared_sigs)) + .iter() + .cloned() + .fold( + v1_rt::rc_empty_map::>(), + |acc: Rc>>, dsig: Rc| { + if emit_map_has(local_func_set.clone(), dsig.name.clone()) { + acc.clone() + } else { + if (dsig.inferred.clone() != None) { + v1_rt::rc_map_insert( + acc.clone(), + dsig.name.clone(), + declared_to_resolved(dsig.clone()), + ) + } else { + acc.clone() + } + } + }, + ) +} + pub fn declared_to_resolved(dsig: Rc) -> Rc { Rc::new(ResolvedFuncSig { name: dsig.name.clone(), @@ -450,7 +477,6 @@ pub fn resolve_func_sigs( items: Rc>>, module_name: String, source_indices: Rc>>, - parent_sigs: Rc>>, ) -> Rc { { let local_func_names = Rc::new({ @@ -477,22 +503,11 @@ pub fn resolve_func_sigs( local_func_set.clone(), source_indices.clone(), ); - let parent_sigs_filtered = Rc::new(v1_rt::map_values(&parent_sigs)) - .iter() - .cloned() - .fold( - v1_rt::rc_empty_map::>(), - |acc: Rc>>, rsig: Rc| { - if emit_map_has(local_func_set.clone(), rsig.name.clone()) { - acc.clone() - } else { - v1_rt::rc_map_insert(acc.clone(), rsig.name.clone(), rsig.clone()) - } - }, - ); + let parent_resolved = + collect_parent_resolved_sigs(declared_sigs.clone(), local_func_set.clone()); topo_resolve_loop( local_func_names.clone(), - parent_sigs_filtered, + parent_resolved, declared_sigs.clone(), call_edges, local_func_set.clone(), diff --git a/src/v2/compiler/05_emit_orchestration.dag b/src/v2/compiler/05_emit_orchestration.dag index a5fe9c1e48d..861e34c317a 100644 --- a/src/v2/compiler/05_emit_orchestration.dag +++ b/src/v2/compiler/05_emit_orchestration.dag @@ -4,10 +4,10 @@ import v2.compiler.emit { emit } import v2.compiler.infer { InferredFacts, InferredTree } import v2.compiler.emit_shell { emit_shell_stmt } import extdeps.languages.bash.program { RawLine } -import v2.workflow.orch_env_lower { lower_env_bindings } import extdeps.communication.medium { Medium } import v2.extdeps.languages.bash { orch_run_emitted_command, + orch_run_target_model, orch_seq2_emitted_command, orch_seq2_target_model, orch_retry_emitted_command, @@ -50,10 +50,23 @@ fn orch_emit_facts_lookup(key: Node) -> Witness { } } -data orch_run_env_reduction_dissolution_trigger: String = "DISSOLVED. orchestration_emit Run path reduces fully onto emit_shell_stmt via lower_env_bindings (orch_env_lower.dag): env-FREE Run lowers to RawLine (proven byte-identical by orch_run_empty_env_delegates_to_shell_emit_holds); env-BEARING Run lowers via lower_env_bindings to a nested EnvPrefixed/EnvUnsetCmd ShellStmt tree over the Run.env FreeMonoid, then delegates to emit_shell_stmt (proven byte-identical + discriminating by orch_run_multi_binding_env_emits_correctly). orch_emit_run_env_welded DELETED as dissolution receipt (#5841 kernel straddle + #5846 bash AST EnvUnsetCmd + lower_env_bindings = all blockers cleared)." +data orch_run_env_reduction_dissolution_trigger: String = "orchestration_emit Run/Step/Retry path reduces onto emit_shell_stmt: an env-FREE Run lowers to a RawLine carrying the command and is delegated to emit_shell_stmt now (single orch_raw, green path, proven byte-identical by orch_run_empty_env_delegates_to_shell_emit). An env-BEARING Run lowers to an EnvPrefixed over env plus a RawLine command. The emit-kernel String<->List-Int-codepoint straddle that blocked this is now FIXED by grounding String construction into its native realization (free_monoid_to_string in v1_interpreter.rs, sibling to Int<->Nat PR 5428; the held bash multi-orch_raw weld fixtures have been deleted as receipt; program_multi_orch_raw_byte_identical_holds is the green-by-execution proof). The RESIDUAL blocker is a DIFFERENT, modeling-side gap, not the kernel: EnvBinding has EnvUnset (env -u NAME) and FreeMonoid is a LIST, but the bash AST EnvPrefixed models only a single name=value prefix with no EnvUnset node, so an env-bearing Run cannot yet be lowered losslessly. It stays on the orch_run_emitted_command single-string render (orch_emit_run_env_welded) until the bash AST grows an env-unset construct and multi-binding nesting. DISSOLVES WHEN the bash EnvPrefixed AST covers EnvUnset + multi-binding env (modeling follow-on landed by gunbc#5846 'Grow bash AST: EnvUnset node + multi-binding env nesting (dissolve orch_emit_run_env_welded)', which depends on this PR's kernel String grounding and dissolves this welded fn outright; no longer kernel-blocked)." fn orch_emit_run(r: Run) -> Outcome> { - emit_shell_stmt(s: lower_env_bindings(env: r.env, command: RawLine { text: r.command })) + match list_head(xs: r.env) { + HeadAbsent => emit_shell_stmt(s: RawLine { text: r.command }) + HeadFound { value: _ } => orch_emit_run_env_welded(r: r) + } +} + +fn orch_emit_run_env_welded(r: Run) -> Outcome> { + emit( + tree: InferredTree { + root: orch_run_emitted_command, + facts: Map { lookup: orch_emit_facts_lookup } + }, + target: orch_run_target_model(r: r) + ) } fn orch_emit_join2(left: String, right: String) -> Outcome> { diff --git a/src/v2/compiler/05_emit_orchestration_stmt_test.dag b/src/v2/compiler/05_emit_orchestration_stmt_test.dag index a73a8429c2a..cf577d87835 100644 --- a/src/v2/compiler/05_emit_orchestration_stmt_test.dag +++ b/src/v2/compiler/05_emit_orchestration_stmt_test.dag @@ -1,6 +1,6 @@ module v2.compiler.emit_orchestration_stmt_test -import v2.compiler.emit_shell { emit_shell_stmt, emit_shell_program, emit_shell_word_raw_scaffold } +import v2.compiler.emit_shell { emit_shell_stmt, emit_shell_program } import extdeps.languages.bash.program { ShellStmt, Command, @@ -12,7 +12,6 @@ import extdeps.languages.bash.program { AndOr, Pipe, EnvPrefixed, - EnvUnsetCmd, WithRedir, Heredoc, AndThen, @@ -160,34 +159,6 @@ test fn stmt_env_prefixed_has_teeth_holds() -> Bool { stmt_differs_from_oracle_of(s: envp_t(), other: EnvPrefixed { name: "TMPDIR", value: Lit { text: "/x" }, command: cmd_true() }) } -fn env_unset_t() -> ShellStmt { - EnvUnsetCmd { name: "TMPDIR", command: cmd_true() } -} - -test fn stmt_env_unset_byte_identical_holds() -> Bool { - stmt_emits_oracle(s: env_unset_t()) -} - -test fn stmt_env_unset_has_teeth_holds() -> Bool { - stmt_differs_from_oracle_of(s: env_unset_t(), other: EnvUnsetCmd { name: "OTHER", command: cmd_true() }) -} - -fn env_multi_binding_t() -> ShellStmt { - EnvPrefixed { - name: "KEY", - value: Lit { text: "val" }, - command: EnvUnsetCmd { name: "TMPDIR", command: cmd_true() } - } -} - -test fn stmt_env_multi_binding_byte_identical_holds() -> Bool { - stmt_emits_oracle(s: env_multi_binding_t()) -} - -test fn stmt_env_multi_binding_has_teeth_holds() -> Bool { - stmt_differs_from_oracle_of(s: env_multi_binding_t(), other: EnvPrefixed { name: "KEY", value: Lit { text: "other" }, command: EnvUnsetCmd { name: "TMPDIR", command: cmd_true() } }) -} - fn heredoc_t() -> ShellStmt { Heredoc { path: Lit { text: "out" }, delimiter: "EOF", body: "hello" } } @@ -384,9 +355,3 @@ test fn redir_stdout_to_stderr_has_teeth_holds() -> Bool { stmt_emit_carried(s: withredir_stdout_to_stderr()) != stmt_emit_carried(s: WithRedir { stmt: cmd_true(), redir: StderrToNull }) } - -test fn emit_shell_word_raw_scaffold_recorded_holds() -> Bool { - string_contains(s: emit_shell_word_raw_scaffold, pattern: "SCAFFOLD") - && string_contains(s: emit_shell_word_raw_scaffold, pattern: "LATENT") - && string_contains(s: emit_shell_word_raw_scaffold, pattern: "DISSOLVES WHEN") -} diff --git a/src/v2/compiler/05_emit_shell.dag b/src/v2/compiler/05_emit_shell.dag index 3eddaa8e8e0..57e3f040f7a 100644 --- a/src/v2/compiler/05_emit_shell.dag +++ b/src/v2/compiler/05_emit_shell.dag @@ -23,8 +23,6 @@ import v2.extdeps.languages.bash { bash_stmt_with_redir_target_model, bash_stmt_env_prefixed_emitted, bash_stmt_env_prefixed_target_model, - bash_stmt_env_unset_emitted, - bash_stmt_env_unset_target_model, bash_stmt_if_emitted, bash_stmt_if_target_model, bash_stmt_heredoc_emitted, @@ -60,7 +58,6 @@ import extdeps.languages.bash.program { AndOr, Pipe, EnvPrefixed, - EnvUnsetCmd, WithRedir, Heredoc, Exit, @@ -156,12 +153,11 @@ fn emit_shell_word(w: ShellWord) -> Outcome> { bind_outcome(o: emit_shell_stmt(s: b), f: fn(m) { emit_word_cmdsubst(body: m.carried) }) - Raw { text: t } => emit_word_pass(spelling: t) + Raw { text: _ } => + outcome_rejected(shell_emit_unsupported(reason: ^emit_shell_word_raw_uninhabited)) } } -data emit_shell_word_raw_scaffold: String = "SCAFFOLD: Raw ShellWord emits verbatim via bash_word_pass (bash_token_orch_raw span, no quoting or escaping). This matches the behavior of the deleted orch_emit_run_env_welded weld, which also emitted env values as verbatim orch_raw tokens. An EnvSet.value with shell metacharacters (space, $, quotes) mis-emits — e.g. FOO=hello world emits as the token 'hello world' so bash parses it as FOO=hello then executes 'world'. LATENT: current env-value consumers are simple CI tokens (no metacharacters), so the gap does not surface in production. DISSOLVES WHEN env-value emission routes through a quoted or escaped-word production, OR EnvSet.value is constrained to a no-metachar token type by construction. Added by #5868." - fn emit_stmt_subshell(body: String) -> Outcome> { emit( tree: InferredTree { @@ -232,16 +228,6 @@ fn emit_stmt_env_prefixed(name: String, value: String, command: String) -> Outco ) } -fn emit_stmt_env_unset(name: String, command: String) -> Outcome> { - emit( - tree: InferredTree { - root: bash_stmt_env_unset_emitted, - facts: Map { lookup: shell_emit_facts_lookup } - }, - target: bash_stmt_env_unset_target_model(name: name, command: command) - ) -} - fn emit_stmt_if(cond: String, body: String) -> Outcome> { emit( tree: InferredTree { @@ -427,10 +413,6 @@ fn emit_shell_stmt(s: ShellStmt) -> Outcome> { emit_stmt_env_prefixed(name: n, value: mv.carried, command: mc.carried) }) }) - EnvUnsetCmd { name: n, command: c } => - bind_outcome(o: emit_shell_stmt(s: c), f: fn(mc) { - emit_stmt_env_unset(name: n, command: mc.carried) - }) WithRedir { stmt: st, redir: rd } => bind_outcome(o: emit_shell_stmt(s: st), f: fn(mst) { bind_outcome(o: emit_redir(r: rd), f: fn(mr) { diff --git a/src/v2/extdeps/languages/bash.dag b/src/v2/extdeps/languages/bash.dag index 6c78cdb5cfc..0e7ee7193c9 100644 --- a/src/v2/extdeps/languages/bash.dag +++ b/src/v2/extdeps/languages/bash.dag @@ -1822,58 +1822,6 @@ fn bash_stmt_env_prefixed_target_model(name: String, value: String, command: Str ) } -fn bash_stmt_env_unset_tokens() -> List { - [ - FixedToken { token_class: ^bash_token_env_unset_prefix }, - BoundToken { token_class: ^bash_token_orch_raw, binding: ^bash_stmt_envp_unset_name }, - FixedToken { token_class: ^bash_token_env_unset_sp }, - BoundToken { token_class: ^bash_token_orch_raw, binding: ^bash_stmt_envp_unset_command } - ] -} - -data bash_stmt_env_unset_emitted: Node = bash_conj_emitted_from_tokens( - tokens: bash_stmt_env_unset_tokens() -) - -fn bash_stmt_env_unset_source_text(name: String, command: String) -> String { - concat("env -u ", concat(name, concat(" ", command))) -} - -fn bash_stmt_env_unset_lex() -> LexRules { - phrase_to_lex_rules(p: phrase([ - span(text: "env -u ", class: ^bash_token_env_unset_prefix), - span(text: " ", class: ^bash_token_env_unset_sp) - ])) -} - -fn bash_stmt_env_unset_translation_rules_node() -> Node { - bash_relation_translation_rules_node( - rule_name: ^bash_relation_rule_stmt_env_unset, - lhs: ^bash_production_stmt_env_unset, - tokens: bash_stmt_env_unset_tokens(), - emitted: bash_stmt_env_unset_emitted - ) -} - -fn bash_stmt_env_unset_binding_spellings(name: String, command: String) -> Map { - map_insert( - map_insert(empty_map(), ^bash_stmt_envp_unset_name, name), - ^bash_stmt_envp_unset_command, command - ) -} - -fn bash_stmt_env_unset_target_model(name: String, command: String) -> TargetModel { - bash_make_bound_tokens_target_model( - rules: bash_stmt_env_unset_translation_rules_node(), - source_literal: ^bash_stmt_env_unset_source_literal, - production_name: ^bash_production_stmt_env_unset, - tokens: bash_stmt_env_unset_tokens(), - source_text: bash_stmt_env_unset_source_text(name: name, command: command), - lex: bash_stmt_env_unset_lex(), - binding_spellings: bash_stmt_env_unset_binding_spellings(name: name, command: command) - ) -} - fn bash_stmt_if_tokens() -> List { [ FixedToken { token_class: ^bash_token_if_kw }, diff --git a/src/v2/lens/realization_vocabulary_containment.dag b/src/v2/lens/realization_vocabulary_containment.dag index 406d7d618da..ffe910b0228 100644 --- a/src/v2/lens/realization_vocabulary_containment.dag +++ b/src/v2/lens/realization_vocabulary_containment.dag @@ -30,8 +30,7 @@ data realization_edge_path_prefixes: List = [ "src/v2/workflow/shell_emit", "src/v2/workflow/orchestration_emit", "src/v2/workflow/orchestration_stmt_emit", - "src/v2/workflow/orchestration_word_emit", - "src/v2/workflow/orch_env_lower" + "src/v2/workflow/orchestration_word_emit" ] data realization_vocab_exception_roster: List = [ diff --git a/src/v2/workflow/orch_env_lower.dag b/src/v2/workflow/orch_env_lower.dag deleted file mode 100644 index 68231b3995d..00000000000 --- a/src/v2/workflow/orch_env_lower.dag +++ /dev/null @@ -1,24 +0,0 @@ -module v2.workflow.orch_env_lower - -import v2.std.orchestration { EnvBinding, EnvSet, EnvUnset } -import v2.std.algebra { FreeMonoid, list_head, list_tail, HeadFound, HeadAbsent, TailFound, TailAbsent } -import extdeps.languages.bash.program { ShellStmt, EnvPrefixed, EnvUnsetCmd, Raw } - -fn lower_env_binding(b: EnvBinding, inner: ShellStmt) -> ShellStmt { - match b { - EnvSet { name: n, value: v } => EnvPrefixed { name: n, value: Raw { text: v }, command: inner } - EnvUnset { name: n } => EnvUnsetCmd { name: n, command: inner } - } -} - -fn lower_env_bindings(env: FreeMonoid, command: ShellStmt) -> ShellStmt { - match list_head(xs: env) { - HeadAbsent => command - HeadFound { value: b } => - match list_tail(xs: env) { - TailAbsent => lower_env_binding(b: b, inner: command) - TailFound { tail: rest } => - lower_env_binding(b: b, inner: lower_env_bindings(env: rest, command: command)) - } - } -} diff --git a/src/v2/workflow/orchestration_emit_test.dag b/src/v2/workflow/orchestration_emit_test.dag index 4aaaaf3de2c..e127b47de87 100644 --- a/src/v2/workflow/orchestration_emit_test.dag +++ b/src/v2/workflow/orchestration_emit_test.dag @@ -8,8 +8,7 @@ import v2.std.orchestration { Do, Pipeline, FailFast, - EnvSet, - EnvUnset + EnvSet } import v2.std.collection { Absent } import v2.std.diagnostic { Accepted, None, Outcome, Rejected } @@ -91,54 +90,7 @@ test fn orch_run_empty_env_delegates_to_shell_emit_holds() -> Bool { } test fn orch_run_env_reduction_trigger_recorded_holds() -> Bool { - string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "DISSOLVED") - && string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "EnvUnsetCmd") - && string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "lower_env_bindings") - && string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "DELETED") -} - -fn orch_run_multi_binding_env() -> Run { - Run { - command: "cargo build", - env: [EnvSet { name: "CARGO_BUILD_JOBS", value: "1" }, EnvUnset { name: "TMPDIR" }], - redirect: Absent, - capture: Absent - } -} - -fn orch_run_multi_binding_env_oracle() -> String { - "CARGO_BUILD_JOBS=1 env -u TMPDIR cargo build" -} - -test fn orch_run_multi_binding_env_emits_correctly_holds() -> Bool { - match orch_emit_run(r: orch_run_multi_binding_env()) { - Accepted { value: src, diagnostics: dg } => - (src.carried == orch_run_multi_binding_env_oracle()) && (dg == None) - Rejected { diagnostics: _ } => false - } -} - -test fn orch_run_multi_binding_env_has_teeth_holds() -> Bool { - match orch_emit_run(r: orch_run_multi_binding_env()) { - Accepted { value: src, diagnostics: _ } => - src.carried != "CARGO_BUILD_JOBS=1 cargo build" - Rejected { diagnostics: _ } => false - } -} - -fn orch_run_env_metachar_value() -> Run { - Run { - command: "cargo build", - env: [EnvSet { name: "FOO", value: "hello world" }], - redirect: Absent, - capture: Absent - } -} - -test fn orch_run_env_metachar_value_emits_verbatim_known_gap_holds() -> Bool { - match orch_emit_run(r: orch_run_env_metachar_value()) { - Accepted { value: src, diagnostics: _ } => - src.carried == "FOO=hello world cargo build" - Rejected { diagnostics: _ } => false - } + string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "EnvPrefixed") + && string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "straddle") + && string_contains(s: orch_run_env_reduction_dissolution_trigger, pattern: "DISSOLVES WHEN") }