diff --git a/dag/gunbc/design_document.dag b/dag/gunbc/design_document.dag index 8708219294c..2c5ddd149fa 100644 --- a/dag/gunbc/design_document.dag +++ b/dag/gunbc/design_document.dag @@ -150,7 +150,7 @@ fn open_threads_blocks() -> List { li(text: "**namespace-only name resolution — the containment tree is the single naming authority (operator-signed 2026-07-06).** Supersedes resolver-graph-major §1c's *import-name-universe* (\"own declarations ∪ direct import lists\" visibility), carrying forward its mechanics unchanged (binding-edge owner, unbound/double-bound = error, patterns-via-scrutinee, no expected-type picker). One structure — **syntactic containment** — induces everything at once: qualified name = nesting position (`T\{a\}` → `a` *is* `T.a`; `T\{a\{a\}\}` → inner *is* `T.a.a`, so Rule-2 no-ambiguity holds by construction), reference = lexical lookup up the ancestor chain (a sibling module is visible, its members projected `node.Symbol`), `.` = projection one level down (field / module-member / variant — one op), and `.` *is* the `std.induction` sub-value relation (descent evidence). So one content-addressed tree grounds THREE consumers — resolution *walks* it, content-addressing *hashes* it, termination reads its *sub-value* edges (§2 Realization: one structure, N consumers). Frontend BUILDS the tree from nesting (pure structure, zero resolution logic); backend WALKS it (lexical up, project down) — no heuristics to tune (the tuning risk = resolving against a flat index instead of the tree). Two governing rules (operator, 2026-07-06): **no dual representation / minimal at every layer; no ambiguity at any layer.** Rides on `SymbolIndex` = \"the containment tree materialized\" (`type-env-single-authority`, lively-raven lane), gated on loyal-heron's scaling receipt before any resolver surgery; `import-scoped` (§1c, today) and `namespace-only-Y` (position-info uniqueness: expected type filters a variant to one, never picks) are two walk-rules over one *fill-agnostic* index (fill = whole tree; policy gates lookup, never fill). Census: 98% of names globally unique → always bare; the residue is v1-seed forks (resolve by subtree, free) + github-style variants ((Y) context-resolves); no consolidate-now forks (the census's 2 same-name flags proved to be a homonym + whole-file v1-seed duplication, not genuine cross-tree §3 forks). Terminal step: delete the `import` grammar + supporting code → `import` becomes a *parse error*, deps derived from `container.member` references (Rule-1 end-state). → [namespace-only resolution design](docs/plans/namespace-resolution-design.md)"), li(text: "**fleet-reconcile spine — one grain-agnostic membership diff (deploy · fleet · session).** ONE `membership_reconcile` reuses `std.change.keyed_two_way_diff` verbatim (no fork): desired set vs observed set keyed by member IDENTITY — Added/Modified → upsert, Removed → teardown-or-refuse, Unchanged → noop (absence of a hunk). A member may be a service/unit/host OR a Session (operator: 'it should all be the same'), so a new member type is a new instantiation of the same fn with its own `(key_of, key_eq, value_eq, ownership_of)` bundle — zero spine change (a Realization, §2; a forked reconcile means the genericity bought nothing). R5 teardown-owned-only is a CONSTRUCTION WALL: a Removed non-owned member yields `MemberTeardownRefused`, which has no effect arm in any apply dispatch, so the bad state is unwritable — never a post-check, never 'assume owned' (the absorbing fallback §5 forbids); ownership-unknown REFUSES too, typed/located/counted. Ownership is extracted as its own single authority (`gunbc.ownership.Ownership`) and `live_deploy`'s `DeploymentStep` re-grounds onto it (pass 2, a declared dissolution trigger — never a second `Owned|Ensured`). Pass 1 (spine + R5 + a synthetic discriminating witness the degenerate apply/retract poles cannot exercise) landed. → [membership-diff reconcile spine design](docs/plans/membership-diff-reconcile-spine-design.md)"), li(text: "**effect grants over namespaces — dissolve `Hermetic | Wet` into (frame × verb × subtree).** The 2-valued `Hermetic | Wet` enum conflates four axes (input closure / replayability, output reach / interference, handler binding, selection eligibility) — a §3 state-space conflation with proto-envelopes already forked (the hand-rolled `workspace_root` containment gate, `std.resources.ResourceHandle`, `AuthScope`, `LiveTreeDisposition`). End shape (§3, cited not minted): an effect target is a position in a containment tree that already exists — filesystem paths, URIs, `/proc`, service-operation paths, and the unifying case **code names themselves** (the containment tree the namespace-resolution lane makes the single naming authority); permission is a grant of (verb × subtree) attached to a `Frame`; admissibility is the namespace prefix relation — the same `⊑` the resolver walks, content-addressing hashes, and termination reads, so effects become the **fourth consumer** of the one containment structure. Dispatch checks the envelope fail-closed (`EffectOutsideGrant`, a typed/located/counted refusal, never a silent widen — §5); *replayable* / *isolated* / `LiveTreeDisposition` become **derived** projections, and `Hermetic | Wet | Record` survive only as named envelope presets, deleted at the end. Frame-containment is graded on the §5 construction/validation axis (`LifecycleByConstruction` vs `LifecycleByConvention`). No code lands from the design doc; the FLAG-A interim is the only near-term consumer. → [effect-namespace-grants design](docs/plans/effect-namespace-grants.md)"), - li(text: "**shell → intent: the intent layer is language-blind (operator-aligned 2026-07-17).** §3/§4 restated for the shell case: the `.dag` intent may not name a target language — a workflow is an ordinary dependency graph over modeled operations (§4: a program is `Node`+`Edge`), and rendering it to bash is a separate, target-parametrized concern, so bash appears exactly where Rust does (its grammar spec `src/v2/extdeps/languages/bash.dag` + the emit rows) and NOWHERE in the intent (operator: *the intent IS the `.dag` graph*; `src/v2/std/orchestration.dag` `Pipeline` is optional sugar for linear-orchestration graphs, never the authority). Two distinct downstream layers, not to be conflated: **emit** (§4 — `emit(intent, Bash)`, one grammar read backward, target as a *parameter*) and **realization** (§2 — pure-spec → host-effect, N transports `LocalShell`/`SshShell`, `dag/gunbc/host_effect_realize.dag`) — emit renders surface, realization effects a host, and realization consumes emit's output. **Invariant (enforceable, not aspirational):** the intent imports no language-construction vocab (`bash_build`, `bash_command_fold_serialize`, the `ShellStmt`/`ShellWord`/`ShellProgram` coproduct, `serialize_bash`) — that vocab is emit-internal; `src/v2/lens/realization_vocabulary_containment.dag` **generalized** with the intent layer in scope, green = *conforms to §3/§4* (the natural `StandingIntent` home). **Residual (2026-07-17, operator-flagged):** two live classes — (1) raw `concat(...)` shell strings (`dag/gunbc/ci_spec.dag` `gunbc_ci_deploy_invoke` — UNTOUCHED by the `bash_build` migration, which only migrated the `ShellStmt`/`program.dag` consumers) and (2) structured-but-still-bash (`src/v2/workflow/floor_diff_observe.dag` `floor_git_diff_unified_stmts`/`floor_serialize_program`) — census counts ~110 raw-`concat` shell fns + ~9 structured sites across ~50 files (bulk in `dag/gunbc/**`; `dag/std`/`src/v2/lens`/`src/v2/compiler` clean), collapsing to ~10 operation families (gunbc-run · source-root flags · git · cargo/rustup/tar · deploy-preflight · live_deploy · install · systemd read-back · githooks · curl) — so it is ~10 operations to model, not 110 problems. **Phases:** 0 sidecar delete (in flight — PR-A/PR-B merged, PR-C/PR-D queued) → 1 complete the agnostic emit (`Pipeline` bounded-poll/`While`, general `Retry` — the gap that pushed workflows to `bash_build`) → 2 model the operations with transports (`git.diff`, the `gunbc run` invocation, unit upsert, readiness-poll — bash-CLI as one handler of N) → 3 migrate the intent off shell → wall green throughout. Flagship: `git.diff` via `floor_diff_observe`, end-to-end intent→emit→realization. Couples to `dag/gunbc/host_effect.dag` `ShellCommand\{script\}` — the same bash-shaped hole one layer down (the fleet-reconcile spine's realization), which should become *run this operation* with bash-rendering downstream, not a deepened `ShellCommand\{script\}`. → [shell → intent design](docs/plans/shell-intent-emit-realization-design.md)"), + li(text: "**shell → intent: the intent layer is language-blind (operator-aligned 2026-07-17).** §3/§4 restated for the shell case: the `.dag` intent may not name a target language — a workflow is an ordinary dependency graph over modeled operations (§4: a program is `Node`+`Edge`), and rendering it to bash is a separate, target-parametrized concern, so bash appears exactly where Rust does (its grammar spec `src/v2/extdeps/languages/bash.dag` + the emit rows) and NOWHERE in the intent (operator: *the intent IS the `.dag` graph*; `src/v2/std/orchestration.dag` `Pipeline` is optional sugar for linear-orchestration graphs, never the authority). Two distinct downstream layers, not to be conflated: **emit** (§4 — `emit(intent, Bash)`, one grammar read backward, target as a *parameter*) and **realization** (§2 — pure-spec → host-effect, N transports `LocalShell`/`SshShell`, `dag/gunbc/host_effect_realize.dag`) — emit renders surface, realization effects a host, and realization consumes emit's output. **Invariant (enforceable, not aspirational):** the intent imports no language-construction vocab (`bash_build`, `bash_command_fold_serialize`, the `ShellStmt`/`ShellWord`/`ShellProgram` coproduct, `serialize_bash`) — that vocab is emit-internal; `src/v2/lens/realization_vocabulary_containment.dag` **generalized** with the intent layer in scope, green = *conforms to §3/§4* (the natural `StandingIntent` home). **Residual (2026-07-17, operator-flagged):** two live classes — (1) raw `concat(...)` shell strings (`dag/gunbc/ci_spec.dag` `gunbc_ci_deploy_invoke` — UNTOUCHED by the `bash_build` migration, which only migrated the `ShellStmt`/`program.dag` consumers) and (2) structured-but-still-bash (`src/v2/workflow/floor_diff_observe.dag` `floor_git_diff_unified_stmts`/`floor_serialize_program`) — census counts ~110 raw-`concat` shell fns + ~9 structured sites across ~50 files (bulk in `dag/gunbc/**`; `dag/std`/`src/v2/lens`/`src/v2/compiler` clean), collapsing to ~10 operation families (gunbc-run · source-root flags · git · cargo/rustup/tar · deploy-preflight · live_deploy · install · systemd read-back · githooks · curl) — so it is ~10 operations to model, not 110 problems. **Phases:** 0 sidecar delete — LANDED (#6831) → **1 complete the agnostic emit — LANDED (#6832):** `While`/`BoundedPoll`/general `Retry` (N-level escalation) emit; production consumer `ci_floor_peak_emit.dag` → 2 model the operations with transports (`git.diff`, the `gunbc run` invocation, unit upsert, readiness-poll — bash-CLI as one handler of N) → 3 migrate the intent off shell → wall green throughout. Flagship: `git.diff` via `floor_diff_observe`, end-to-end intent→emit→realization. Couples to `dag/gunbc/host_effect.dag` `ShellCommand\{script\}` — the same bash-shaped hole one layer down (the fleet-reconcile spine's realization), which should become *run this operation* with bash-rendering downstream, not a deepened `ShellCommand\{script\}`. → [shell → intent design](docs/plans/shell-intent-emit-realization-design.md)"), li(text: "**srvN build-cache provisioning on host-standup subsumption spine.** Legacy `ctrl-sccache.service` is misconfigured; `ci_release_build_script()` absorbing fallback (`CARGO_BUILD_JOBS=1` then `-u RUSTC_WRAPPER`) masks the deficit (§5). Design anchor: provision sccache as a host-effect ensure on the assimilation spine (P1b after `RunnerDeploySlot`), generalizing srv3's `WorkflowEnsureActuatorToolchain` to srvN; `ProvisionBuildCache { catalog_id }` routes through `host_effect_apply`; verdict fold is `ProvisionConverged | ProvisionRefused` only. STEP 2 (fallback removal) operator-sequenced after T4 live read-back. → [srvN build-cache provisioning design](docs/plans/srvn-buildcache-provisioning-design.md)"), ]), ] diff --git a/dag/gunbc/plans/shell_emission_model.dag b/dag/gunbc/plans/shell_emission_model.dag index d88f8cc0927..dd961931fa5 100644 --- a/dag/gunbc/plans/shell_emission_model.dag +++ b/dag/gunbc/plans/shell_emission_model.dag @@ -45,7 +45,7 @@ fn shell_emission_model_body() -> List { li(text: "**Slice 4 — tail consumers (pre-runtime residue):** `bmc_token_federation` (two `Do\{Run\}` rows — slice-0 machinery suffices) → `ci_workflow` inline `RunStep`s (case/`uname` → model as `TargetArchitecture`; cross-link ROADMAP §1 `1-inline-shell-defork`) → githooks as a **thin shim** (ensure-built + exec `claim_batch --pre-push` with stdin passed through; the case-rosters/arrays/while-read stdin parse move into the binary)."), ]), h2(text: "6. Sidecar dissolution (parallel)"), - p(text: "**LANDED (#6831, Phase 0):** `dag/extdeps/languages/bash/program.dag` (`ShellProgram`/`serialize_bash`) deleted; the vacuous bash-program importer-count ratchet is pruned with it (resolve fails before the ratchet could fire). Remaining arc work is intent-layer shell→`emit(intent, Bash)` migration (Categories A–C in [shell-intent-emit-realization-design.md](shell-intent-emit-realization-design.md)), not sidecar restoration."), + p(text: "**LANDED (#6831, Phase 0):** `dag/extdeps/languages/bash/program.dag` (`ShellProgram`/`serialize_bash`) deleted; the vacuous bash-program importer-count ratchet is pruned with it (resolve fails before the ratchet could fire). Remaining arc work is intent-layer shell→`emit(intent, Bash)` migration (Categories A–C in [shell-intent-emit-realization-design.md](shell-intent-emit-realization-design.md)), not sidecar restoration. **Phase 1 LANDED (#6832):** `While`/`BoundedPoll`/general `Retry` emit — see `roadmap_authority.dag` `6-shell-intent-phase1`."), p(text: "Tracked in [emission-ingestion-inverse.md](emission-ingestion-inverse.md) / `emission_ingestion_inverse.dag` — **cross-link only, do not duplicate** the roster here."), h2(text: "7. Named residue / dissolution triggers"), ul(items: [ diff --git a/dag/gunbc/roadmap_authority.dag b/dag/gunbc/roadmap_authority.dag index 189f880908b..dea7d07a85d 100644 --- a/dag/gunbc/roadmap_authority.dag +++ b/dag/gunbc/roadmap_authority.dag @@ -216,10 +216,11 @@ fn section_2() -> RoadmapSection { authored(id: "1-g3-caps", done: false, content: "**G3 cgroup caps derived + reconciled** — `TasksMax`/`MemoryMax` are host-set by hand and only *read* live; derive from the budget tree + reconcile via converge so a hand-edit reds."), authored_doc(id: "1-g1-placement", done: false, content: "**G1 placement** — job→host is GitHub-native, demand-blind, first-idle → heavy runs co-reside while the other host idles. LIVE again since #6111 put required CI back on the fleet; the admission model above is its demand side.", path: "docs/plans/compute-envelope-model.md"), authored_doc(id: "6-shell-emission", done: false, content: "**shell emission model** — `emit(intent, Bash)`; scoped by the bash-minimization rule (operator 2026-07-03): bash is emitted only into foreign executors + bootstrap windows; where the runtime is present, effects are typed argv/REST or binary-interpreted plans (the tail slices are shelved).", path: "docs/plans/shell-emission-model.md"), - derivable(id: "2-debash-orchestration", prs: [6106], title: "**orchestration emission → agnostic registry dispatch**", description: "— de-bash the compiler stage (in flight)"), + derivable(id: "2-debash-orchestration", prs: [6106, 6832], title: "**orchestration emission → agnostic registry dispatch**", description: "— de-bash the compiler stage (#6832: While/BoundedPoll/general Retry emit landed)"), owned(rn: authored_wi(id: "2-emit-partition", done: false, content: "**emit partition cleanup — shell/json/yaml as grammar rows, zero language knowledge in the wrong layer** *(operator directive, 2026-07-01)* — one grammar per language, read in both directions (DESIGN §4); emission is rows in `extdeps/languages/`, never stage code. Four leftovers: (a) `src/v2/compiler/05_emit_orchestration.dag` still carries orchestration→bash lowering IN the compiler stage (the retry 2-level expansion, seq-join) — the #6106 registry dispatch is the exit mechanism; the former env-weld half is DISCHARGED (#5868 grew the bash AST `EnvUnset` + multi-binding `EnvPrefixed`; #6137 dissolved `orch_emit_run_env_welded`); the stage keeps ONE agnostic fold. (b) the bash grammar itself is FORKED across trees — `dag/extdeps/languages/bash/` vs `src/v2/extdeps/languages/bash.dag`+`bash_command_fold.dag` — the same §3 class as the `extdeps.shell` fork (hotfixed #6112), one authority must win. (c) JSON has no grammar authority: N hand-rolled emitters (`roadmap_spawner`'s `json_escape`/`json_str` is a marked scaffold dissolving to `std.primitives.to_json`; `bmc_onboard` and `tailscale_acl_emit` carry their own). (d) YAML: `ci_yaml_emit`/`ci_yaml_validate`/`gha_yaml_fold_pilot` sit in `dag/gunbc/` (workflow layer) while the fold pilot is `src/v2/extdeps/languages/gha_workflow_yaml_fold.dag` — YAML-the-language rows belong in `extdeps/languages/` with GHA-workflow as rows on top, retiring the `yaml_check` Rust scaffold once parse is grammar-owned. The construction wall that makes the splice class unwritable (a string literal is an ATOM, never a COMPOSITION — joins live in `extdeps/languages` rows, never workflow code): [no-smuggled-programs wall dissolved with program.dag FULL DELETE — shell → intent design](docs/plans/shell-intent-emit-realization-design.md).", intricacy: IntricacyHigh, volume: VolumeLarge, repo: "gunbc"), owner: "dispatched 2026-07-02 · adhoc-2040cdfe-46b"), sign(rn: authored_doc(id: "6-shell-slice0", done: true, content: "**slice 0 — CI EAGAIN-retry** ✓ (#6467) — `ci_cargo_eagain_retry_core` → `emit(Retry, Bash)`; byte-oracle = committed `ci.yml`", path: "docs/plans/shell-emission-model.md"), s: signed(by: "operator", works: true, scope_equivalent: true, as_expected: true)), sign(rn: authored_doc(id: "6-shell-slice1", done: true, content: "**slice 1 — control-flow emit (census-scoped)** ✓ (#6475; tier-2 Procedure/Let band #6566) — the `If` band only (`orch_emit_step::If` + else + condition forms + pipes/cmdsubst/`$?`/AndOr/redirect/env words, byte goldens); `For`/`While` NOT in scope — the pre-runtime census (2026-07-03) found zero pre-runtime sites needing them", path: "docs/plans/shell-emission-model.md"), s: signed(by: "operator", works: true, scope_equivalent: true, as_expected: true)), + authored_doc(id: "6-shell-intent-phase1", done: true, content: "**shell→intent Phase 1 — agnostic orchestration emit** ✓ (#6832) — `While`/`BoundedPoll`/general `Retry` (N-level escalation) emit via `05_emit_orchestration` + bash grammar rows; byte goldens `orchestration_*_emit_test`; production consumer `ci_floor_peak_emit.dag`", path: "docs/plans/shell-intent-emit-realization-design.md"), authored_doc(id: "6-shell-slice2", done: false, content: "**slice 2 — converge thin-run** *(IN FLIGHT 2026-07-14 — FLAGs 2a(i)/2b/2c operator-signed, keystone worker dispatched; [census](docs/plans/shell-to-dag-residual-census-and-arc-completion.md) §2)* — fleet_converge steady-state moves into the binary as a typed plan via `apply()` (models `EmitArtifactThenThinRun`); emitted bash shrinks to the fresh-standup/self-repair bootstrap fragment + a thin invocation line (supersedes 'emit the whole `.github/fleet-converge.sh`')", path: "docs/plans/shell-emission-model.md"), authored_doc(id: "2-host-effect-phases", done: false, content: "**`apply()` Phases B–F** — B `host_exec`→`apply()` (gated on srv3 OsInstalled) · C Redfish live (partially via #6097) · D converge lane = EmitArtifactThenThinRun handler (first ctrl LOC deleted) · E pull-mode self-converge (autoinstall plants the on-host agent; the push star retires) · F decom. Phase A landed (#5756).", path: "docs/plans/host-effect-orchestration.md"), owned(rn: authored_wi(id: "2-temporal-effect-spine-a", done: false, content: "**temporal-effect-spine-a — temporal realization spine (T1 vocabulary)** *(operator 2026-07-02; not a workflow engine)* — `std.temporal_effect`: durable facts + `plan_next_step_from_prior_receipt_and_lease` (single prior+lease; list fold is consumer-side); 🟡 markers on stringly receipt labels + derived step id. RED: approval/lease/read-back gates. **Non-goals:** live mutation, DB, scheduler. **Accept (T1):** witness suite green.", intricacy: IntricacyHigh, volume: VolumeSmall, repo: "gunbc"), owner: "dispatched 2026-07-02 · zesty-bat-588"), @@ -254,6 +255,7 @@ fn section_2() -> RoadmapSection { RoadmapEdge { child: nid(s: "2-runner-allocation-v0"), parent: nid(s: "2-host-admission") }, RoadmapEdge { child: nid(s: "2-debash-orchestration"), parent: nid(s: "6-shell-emission") }, RoadmapEdge { child: nid(s: "2-emit-partition"), parent: nid(s: "2-debash-orchestration") }, + RoadmapEdge { child: nid(s: "6-shell-intent-phase1"), parent: nid(s: "6-shell-emission") }, RoadmapEdge { child: nid(s: "6-shell-slice0"), parent: nid(s: "6-shell-emission") }, RoadmapEdge { child: nid(s: "6-shell-slice1"), parent: nid(s: "6-shell-slice0") }, RoadmapEdge { child: nid(s: "6-shell-slice2"), parent: nid(s: "6-shell-slice1") }, diff --git a/docs/plans/shell-intent-emit-realization-design.md b/docs/plans/shell-intent-emit-realization-design.md index 3042d96b4ae..709ab650172 100644 --- a/docs/plans/shell-intent-emit-realization-design.md +++ b/docs/plans/shell-intent-emit-realization-design.md @@ -78,8 +78,8 @@ Legitimate and **excluded** from the residual: `src/v2/extdeps/languages/bash.da The headline is **"no language in the intent,"** with the sidecar delete demoted to Phase 0. -0. **Sidecar delete** (in flight — PR-A/PR-B merged, PR-C migrate-last-consumers + PR-D trivial-delete queued). Removes the `serialize_bash`/`RawLine` string-smuggling vector and consolidates bash on the one grammar. Necessary, not sufficient. -1. **Complete the agnostic emit.** `src/v2/std/orchestration.dag` emit has holes (bounded-poll/`While` rejected, `Retry` hardcoded to two levels) — the gap that pushed workflows to reach for `bash_build`. Close it so the intent graph can express what workflows need. +0. **Sidecar delete** — LANDED (#6831, Phase 0). Removes the `serialize_bash`/`RawLine` string-smuggling vector and consolidates bash on the one grammar. Necessary, not sufficient. +1. **Complete the agnostic emit — LANDED (#6832, 2026-07-18).** `While`/`BoundedPoll`/`Retry` (N-level escalation chain) emit via `05_emit_orchestration` + bash grammar rows; byte goldens in `orchestration_while_emit_test` / `orchestration_bounded_poll_emit_test` / `orchestration_retry_emit_test`; production consumer `ci_floor_peak_emit.dag` (cgroup locate `While`). 2. **Model the operations with transports.** `git.diff`, the `gunbc run` invocation, unit upsert, package fetch, readiness-poll — each a §3 operation shape in `extdeps/**` with the bash-CLI as one handler. 3. **Migrate the intent off shell.** Rewrite `gunbc_ci_deploy_invoke`, `floor_diff_observe`, `build_step_emit`, `live_deploy`, CI to build modeled-operation graphs; `bash_build` and raw `concat`-shell vanish from the intent layer. 4. **Wall green with the intent layer in scope** — the generalized `realization_vocabulary_containment`, as the standing acceptance criterion throughout.