Dsl roadmap

TODO/README.md

@@ -1,64 +1,77 @@
-# TODO — Active Plan Index
-
-Active and planned work items live in this folder. When a plan is complete,
-move it to `TODO/TODONE/`.
-
-Last reconciled: 2026-02-14
-
-## Active Plans
-
-| Plan | Status | Notes |
-|------|--------|-------|
-| `TODO/TODO_URGENT_logging_consolidation.md` | Active | CI/local display + logging parity and failure-first output hardening |
-| `TODO/TODO_hacks.md` | Active | Consolidated runtime/design debt tracker |
-| `TODO/TODO_gcp_infra_parity.md` | In Progress | Primary implementation checklist for GCP infra parity |
-| `TODO/TODO_credential_lifecycle.md` | Draft | Architecture reset for auth/credential lifecycle |
-| `TODO/TODO_workflow_audit.md` | Draft | Workflow consolidation, purity/resource, parallelization roadmap |
-| `TODO/consolidation.md` | Ongoing | Broad consolidation + remaining design-dependent tasks |
-| `TODO/llm-code-review-pipeline.md` | V0 complete, Track 1 open | Resource abstraction track still pending |
-| `TODO/design-codegen-quality.md` | Active (ongoing concern) | Generated code quality and backend idiom coverage |
-| `TODO/TODO_transport_dag_migration.md` | Draft | Recommended migration plan, not implemented end-to-end |
-| `TODO/TODO_testgen_seed_policy_postmortem.md` | Partially complete | Core fix landed; follow-up hardening still open |
-
-## Source Of Truth Notes
-
-- For GCP infra execution progress, treat `TODO/TODO_gcp_infra_parity.md` as
-  the canonical tracker. `docs/design/gcp-service-modeling.md` is architecture
-  reference.
-- For completed work, prefer links under `TODO/TODONE/`; stale `TODO/...`
-  references should be updated during doc edits.
-- Hack debt is primarily tracked in `TODO/TODO_hacks.md`; root-level
-  `TODO_hacks` contains additional historical notes that may still have open
-  items.
-
-## Recently Moved To TODONE
-
-- `TODO/TODONE/TODO_ci_timeout_fermi.md` (moved 2026-02-14)
-- `TODO/TODONE/TODO_remove_disallowed_methods_script.md` (moved 2026-02-14)
-
-## Plan Template
+# TODO — DSL Program Index
+
+This folder is now organized around the DSL adoption program.
+Primary roadmap reference: `docs/design/v4/dsl-roadmap.md`.
+**Consolidated execution plan**: [`docs/design/v4/consolidated-worker-plan.md`](../docs/design/v4/consolidated-worker-plan.md) — unified dependency DAG, wave decomposition, and task assignments across all tracks.
+
+Last reconciled: 2026-02-18
+
+## Execution Order
+
+1. Track A: DSL core compiler capabilities
+2. Track B: DSL migration targets (move existing Rust DAG graphs to `.dag`)
+3. Track C: modeling foundations needed for non-fragile DSL migration
+4. Track D: runtime/test hardening required for confident rollout
+5. Track E: domain parity and adjacent programs
+6. Track F: general debt ledger
+
+## Track Definitions
+
+| Track | Purpose |
+|------|---------|
+| A — DSL Core | Language/compiler/runtime features needed by the roadmap |
+| B — Migration Targets | Existing workflows that should be rewritten in DSL now |
+| C — Modeling Foundation | Canonical models (platform, env, transport, composition) that remove stringly/manual wiring |
+| D — Runtime/Test Hardening | Logging, testgen, codegen quality, and execution safety for DSL-generated flows |
+| E — Domain Parity | Product/domain workstreams that should become DSL consumers over time |
+| F — Debt Ledger | Generic cleanup and fallback debt not tied to one feature |
+
+## Active Docs By Track
+
+| Doc | Track | DSL Alignment | Status |
+|-----|-------|---------------|--------|
+| `TODO/TODO_URGENT_dsl_migration.md` | B | Primary migration backlog | Active |
+| `TODO/TODO_workflow_audit.md` | B | Migration inventory + sequencing input | Draft |
+| `TODO/TODO_URGENT_anemic_modeling_audit.md` | C | Cross-cutting model consolidation | Active |
+| `TODO/TODO_URGENT_browser_modeling.md` | C | Platform/environment modeling prerequisite | Active |
+| `TODO/TODO_URGENT_platform_toolchain_modeling.md` | C | Target/platform/toolchain canonicalization | Active |
+| `TODO/TODO_transport_dag_migration.md` | C | Bring transport executor behavior into DAG/testing model | Draft |
+| `TODO/TODO_URGENT_logging_consolidation.md` | D | Execution observability hardening for generated workflows | Active |
+| `TODO/TODO_testgen_seed_policy_postmortem.md` | D | Testgen semantic input correctness | Partially complete |
+| `TODO/design-codegen-quality.md` | D | Generated code idiom/IR quality | Active |
+| `TODO/TODO_credential_lifecycle.md` | E | Credential/service modeling for DSL consumers | Draft |
+| `TODO/TODO_gcp_infra_parity.md` | E | Domain parity backlog (DSL consumer target) | In Progress |
+| `TODO/llm-code-review-pipeline.md` | E | Adjacent DAG pipeline architecture | V0 complete, follow-up open |
+| `TODO/consolidation.md` | F | Generic consolidation backlog | Ongoing |
+| `TODO/TODO_hacks.md` | F | Fallback/debt register | Active |
+
+## Conventions
+
+- Every active TODO doc should include:
+  - `Status`
+  - `Date` (or status date)
+  - `DSL Alignment`
+  - `Track`
+- Use `TODO_URGENT_*.md` only for blockers or high-priority prerequisites.
+- When complete, move docs to `TODO/TODONE/` and note completion date.
+- If a doc is superseded, keep a short pointer at the top to the new source.
+
+## Short Template
 
 ```markdown
-# [Feature Name]
+# [Title]
 
-**Status**: Draft | In Progress | Completed
+**Status**: Draft | Active | In Progress | Completed
 **Date**: YYYY-MM-DD
+**DSL Alignment**: [one line]
+**Track**: A | B | C | D | E | F
 
 ## Goal
 
-[What we're trying to achieve]
-
-## Design
-
-[Architecture, data models, diagrams]
+[Outcome]
 
 ## Tasks
 
 - [ ] Task A
 - [ ] Task B
-- [ ] Task C
-
-## Notes
-
-[Implementation notes, decisions made]
 ```

TODO/TODO_URGENT_anemic_modeling_audit.md

@@ -3,6 +3,8 @@
 **Status**: Active
 **Date**: 2026-02-14
 **Priority**: High
+**DSL Alignment**: DSL foundation via cross-cutting model consolidation
+**Track**: C — Modeling Foundation
 
 ## The Problem Pattern
 

TODO/TODO_URGENT_browser_modeling.md

@@ -1,8 +1,14 @@
 # URGENT: Cross-Platform Browser Open Modeling
 
+**Status**: Active
+**Date**: 2026-02-18
+**Priority**: High
+**DSL Alignment**: DSL foundation via environment-aware platform modeling
+**Track**: C — Modeling Foundation
+
 ## Problem
 
-Browser opening is currently implemented inline in `gunbc-dag/src/dag_viz/graph.rs` (`execute_open_browser`, lines 556-597). It handles WSL/macOS/Linux but:
+Browser opening is currently implemented inline in `gunbc-dag/src/dag_viz/graph.rs` (`execute_open_browser`, around lines 451-486). It handles WSL/macOS/Linux but:
 
 1. **Not shared** -- only usable by dag-viz, not by other tools
 2. **Platform enum is incomplete** -- `lib/tools/deps/src/platform.rs` has `Platform::Linux` but does not distinguish WSL from native Linux
@@ -46,4 +52,4 @@ fn execute_open_browser(inputs: HashMap<String, Value>) -> Result<HashMap<String
 - gunb.ai platform modeling
 - the-gunbai environment detection
 - Current `Platform` enum: `lib/tools/deps/src/platform.rs`
-- Current inline implementation: `gunbc-dag/src/dag_viz/graph.rs:556-597`
+- Current inline implementation: `gunbc-dag/src/dag_viz/graph.rs:451`

TODO/TODO_URGENT_dsl_migration.md

@@ -1,14 +1,36 @@
 # URGENT: DSL Migration Checklist
 
+**Status**: Active
+**Date**: 2026-02-17
+**Last reconciled**: 2026-02-18
+**Priority**: High
+**DSL Alignment**: Primary DSL migration backlog
+**Track**: B — Migration Targets
+
 Hand-rolled patterns that should migrate to daglang as the compiler matures.
 
+## Prerequisite: Type-Dispatch Boilerplate Elimination
+
+> **See**: `TODO/TODO_URGENT_type_dispatch_boilerplate.md` for full audit and implementation plan.
+
+The "Ready Now" migrations are blocked by **~1,350 lines of type-dispatch boilerplate**
+(15 union enums, 16 `From` impls, 10 converter functions) that exist solely to satisfy
+`Dag<T: Executable + Clone + Send + 'static>`. Deleting manual `graph.rs` files requires
+replacing them with DSL-compiled `Dag<DynOp>`, which first requires introducing `DynOp` —
+a type-erased `Arc<dyn Executable + Send + Sync>` wrapper in `core/exec`.
+
+Without `DynOp`, each migrated module would still need its own `GraphOp` union enum and
+converter functions, preserving the boilerplate the DSL was meant to eliminate.
+
+**Fix**: `DynOp` + central resolver (~300 lines added) → delete ~5,650 lines of boilerplate.
+
 ## Ready Now (DSL has the primitives)
 
-- [ ] **Pragma graphs** (`gunbc-dag/src/bin/pragma/graph.rs`) — 3 parallel content
+- [ ] **Pragma graphs** (`gunbc-dag/src/pragma/graph.rs`) — 3 parallel content
       upsert chains. Express as `pattern` invocations with service calls.
 - [ ] **Transport triplets** (all binaries) — prepare/execute/parse 3-node pattern.
       DSL already supports via service call lowering.
-- [ ] **Codegen graph** (`gunbc-dag/src/bin/codegen/graph.rs`) — staged pipeline:
+- [ ] **Codegen graph** (`gunbc-dag/src/codegen/graph.rs`) — staged pipeline:
       exists check → conditional codegen → stamp. DSL `if` in `func` bodies.
 - [ ] **Conditional execution / skip semantics** — content upsert "compare" step
       skips write when content matches. Needs `[skip_if]` or equivalent DSL syntax.
@@ -19,7 +41,7 @@ Hand-rolled patterns that should migrate to daglang as the compiler matures.
       loop with timer ticks. Needs reactive/streaming DSL primitives (`observe events`,
       `every 80ms`). Rendering IR exists (`Frame`, `FrameRenderer`, `OutputMedium`)
       but no DSL construct generates event loops yet.
-- [ ] **Testgen dynamic targets** (`gunbc-dag/src/bin/testgen_dag/graph.rs`) — N
+- [ ] **Testgen dynamic targets** (`gunbc-dag/src/testgen_dag/graph.rs`) — N
       upsert chains, one per `DagSpecDef` discovered via inventory. Needs
       compile-time metaprogramming or inventory integration in DSL.
 - [ ] **Makegen tool registry** — procedural target generation from `#[tool_target]`
@@ -34,7 +56,7 @@ Hand-rolled patterns that should migrate to daglang as the compiler matures.
 | Syntax (types, fn, func, pattern, service, resource, interface, pipeline) | Stable |
 | Lowering to GraphIR | Solid — patterns expand, services → triplets, resources → acq nodes |
 | Type system (records, sums, interfaces, provider resolution) | Working |
-| Pragmatic use (real tools using .dag files) | **0%** — no binary uses DSL yet |
+| Pragmatic use (real tools using .dag files) | In progress — workspace tool composition discovers `dsl/tools/*.dag`; legacy Rust DAG implementations still exist for execution paths/parity |
 
 The "Ready Now" items are the highest-ROI migration targets — pragma especially,
 since it's 3 identical upsert chains that map directly to a `pattern` invocation.

TODO/TODO_URGENT_logging_consolidation.md

@@ -3,6 +3,8 @@
 **Status**: Active
 **Date**: 2026-02-13
 **Priority**: High
+**DSL Alignment**: Runtime observability hardening for DSL-generated workflows
+**Track**: D — Runtime/Test Hardening
 
 ## Motivating Incident: CI Log Explosion (2026-02-13)
 

TODO/TODO_URGENT_platform_toolchain_modeling.md

@@ -0,0 +1,110 @@
+# URGENT: Platform + Toolchain Modeling Gaps (Linux / GNU / QEMU)
+
+**Status**: Active  
+**Date**: 2026-02-18  
+**Priority**: High
+**DSL Alignment**: Canonical platform/target/toolchain model required for DSL portability
+**Track**: C — Modeling Foundation
+
+## Short Answer To The Variant Question
+
+Today, these variants are **not** modeled thoroughly:
+
+- `linux` is modeled in multiple incompatible ways
+- `gnu` (ABI/env in target triples) is mostly not modeled at all
+- `qemu` exists as hardcoded command strings, not as a first-class runtime/emulator concept
+
+## Problem Pattern
+
+The codebase currently has **fragmented platform models** plus **stringly-typed toolchain/runtime branches**.  
+This creates repeated logic and makes it hard to add a new variant without touching many files.
+
+## Fragmentation Map (Current State)
+
+1. DSL platform enum: `dsl/std/types.dag` (`type Platform = Linux | MacOS | Windows`)
+2. Deps runtime platform enum: `lib/tools/deps/src/platform.rs` (`Linux | Macos | Windows | Unknown`)
+3. Tool satisfiability platform model: `core/ir/src/transport/tool.rs` (`PlatformDef`, `PlatformRegistry`, `linux/ubuntu/debian/alpine/macos`)
+4. CI runner models:
+   - `core/ir/src/transport/github_actions.rs` (`RunnerImage` with runner labels + tools, no explicit os/arch fields)
+   - `core/ir/src/transport/ci/runner.rs` (`Runner` trait with string ids/tools)
+5. Codegen target model: `core/daglang/daglang-driver/src/lib.rs` (`CodegenTarget = Rust|Go|C|Mips`) models language backend, not platform/ABI/runtime
+
+## Critical Gaps
+
+- [ ] **No first-class target-triple model (`arch-vendor-os-env`)**
+  - Evidence: `CodegenTarget` only captures backend language in `core/daglang/daglang-driver/src/lib.rs`.
+  - Impact: cannot represent `x86_64-unknown-linux-gnu` vs `x86_64-unknown-linux-musl` without string conventions.
+
+- [ ] **`gnu` / ABI layer is missing**
+  - Evidence: no shared enum/type for `gnu`, `musl`, `msvc`; platform enums stop at OS.
+  - Impact: ABI-sensitive install/build/runtime logic stays ad-hoc.
+
+- [ ] **`qemu`/emulator is not modeled as execution environment**
+  - Evidence: MIPS parity path hardcodes `mips-linux-gnu-as`, `mips-linux-gnu-ld`, `qemu-mips` in `core/daglang/daglang-cli/tests/codegen_parity.rs`.
+  - Impact: emulator support cannot be reused or reasoned about by tool/resource planning.
+
+- [ ] **Environment layer is missing (Native vs WSL vs Container vs CI vs Emulator)**
+  - Evidence: WSL/macOS/Linux branching is inline in `gunbc-dag/src/dag_viz/graph.rs` (`execute_open_browser`).
+  - Impact: every feature needing environment-aware behavior repeats custom detection/branching.
+
+- [ ] **Platform IDs are stringly-typed in install modeling**
+  - Evidence:
+    - `lib/tools/deps/src/manifest.rs` uses `HashMap<String, PlatformInstall>`
+    - `core/ir/src/transport/github/cli.rs` returns `Vec<(&str, InstallMethod)>`
+    - `lib/tools/deps/src/tool_upsert.rs` has a hardcoded PM→platform mapping marked as simplified
+  - Impact: no compile-time guarantees around supported platform keys.
+
+- [ ] **Path resolution bypasses shared platform model**
+  - Evidence: `lib/transport/src/cli.rs` branches directly on `which` vs `where`.
+  - Impact: host-platform behaviors are not centralized.
+
+- [ ] **Generated test mocks hardcode linux defaults**
+  - Evidence: `core/codegen/src/testgen/codegen.rs` maps `"Platform"` mocks to `"linux"`.
+  - Impact: generated tests under-exercise platform variant behavior.
+
+- [ ] **DSL layer currently encodes platform behavior as fixed shell commands**
+  - Evidence: `dsl/tools/dag_viz.dag` browser service is `@shell(["xdg-open", "{path}"])`.
+  - Impact: cross-platform support in DSL authoring remains non-portable.
+
+## What To Borrow From `../the-gunbai`
+
+- `../the-gunbai/crates/gunbai-integrations-contracts/src/understanding/rust_targets.rs`
+  - first-class target-triple constants and mapping helpers
+- `../the-gunbai/crates/gunbai-integrations-contracts/src/understanding/platform.rs`
+  - explicit OS/arch detection + normalization assumptions/unknowns
+- `../the-gunbai/crates/gunbai-integrations-contracts/src/understanding/github_actions_runner.rs`
+  - structured runner spec (`os`, `distro`, `version`) instead of raw labels only
+
+## Canonical Model Direction
+
+- [ ] Introduce one shared platform model in `core/ir` and consume it everywhere:
+  - `Arch`, `Vendor`, `Os`, `AbiEnv` (`gnu|musl|msvc|...`)
+  - `TargetTriple { arch, vendor, os, env }`
+  - `ExecutionEnv` (`Native`, `Wsl`, `Container`, `Ci`, `Emulator`)
+  - `RuntimePlatform { host: TargetTriple, env: ExecutionEnv }`
+
+- [ ] Model toolchain components as resources/tools, not command literals:
+  - assembler, linker, runtime/emulator (`qemu-*`)
+
+- [ ] Make install/run resolution data-driven from the canonical model:
+  - no free-form platform string keys in manifests/registries
+
+## Phased Implementation Checklist
+
+### Phase 1: Foundation Types
+
+- [ ] Add canonical platform/target/env types in `core/ir` (single source of truth)
+- [ ] Add parsing/formatting helpers for target triples and env variants
+- [ ] Add compatibility adapters from existing enums (`deps::Platform`, DSL platform type)
+
+### Phase 2: Highest-ROI Migrations
+
+- [ ] Replace hardcoded MIPS assembler/linker/qemu strings with modeled toolchain resources
+- [ ] Replace inline browser open branching with environment-aware resolver utility
+- [ ] Switch deps install and GH install platform keys to typed platform IDs
+
+### Phase 3: DSL + Testgen Alignment
+
+- [ ] Align DSL `Platform`/`CodegenTarget` vocabulary with canonical types
+- [ ] Remove linux-hardcoded mock defaults in testgen and generate per-platform variants
+- [ ] Add conformance tests for `linux-gnu` vs other env/ABI variants and qemu executor selection