docs(e2e): reconcile scenario migration docs

test/e2e-scenario/docs/MIGRATION.md

@@ -1,22 +1,43 @@
 <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
 <!-- SPDX-License-Identifier: Apache-2.0 -->
 
-# E2E migration tracking
+# E2E scenario migration notes
 
-Migration state for the scenario-based E2E framework is intentionally tracked
-outside the repository, in GitHub issues and pull requests. This file documents
-that policy so the repo does not retain stale per-script checklists, migrated
-counts, or ownership tables after the work moves on.
+This file records the current migration model for contributors. It is not the
+source of truth for per-domain status. Mutable migration state is tracked
+outside the repository in GitHub issues and pull requests so reviewers can
+discuss, update, and close work in one place.
 
-## Where to track current work
+## Current migration state
 
-- Use issue #3588 as the parent epic for the layered E2E scenario model.
-- Use the active audit-coverage phase issues #4347 through #4357 for current
-  domain-by-domain implementation, review, and reconciliation state.
-- Use issue #4378 for the setup-scenario alias cleanup that bridges layered
-  `test_plans` to friendly scenario IDs.
-- Use the pull request for each change as the evidence record for what landed,
-  what was deferred, and what follow-up issues remain.
+The scenario E2E migration is in a hybrid phase:
+
+- typed scenario builders drive scenario workflow fan-out and dry-run plans;
+- product-facing `NemoClawInstance` manifests describe desired setup and
+  onboarding state;
+- YAML metadata still drives the shell scenario runner and live suite
+  resolution;
+- legacy `test/e2e/test-*.sh` scripts still provide most live nightly and
+  platform coverage.
+
+Do not assume legacy scripts are deletion-ready just because a scenario or suite
+name exists. The final reconciliation phase must show either evidence-complete
+coverage or an explicit audit amendment before legacy executable tests are
+removed.
+
+## Active issue tracking
+
+Use these GitHub issues for status and follow-up work:
+
+| Issue | Purpose |
+| --- | --- |
+| #3588 | Parent architecture epic for layered / hybrid scenario E2E |
+| #4347–#4356 | Domain-specific audit-coverage phases |
+| #4357 | Final audit reconciliation, placeholder cleanup, and deletion-readiness review |
+| #4378 | Friendly `setup_scenarios` aliases for layered test plans |
+
+If a migration discovery needs durable tracking, add it to the relevant issue or
+open a focused child issue. Avoid adding long-lived checklists here.
 
 ## What belongs in the repo
 
@@ -31,8 +52,48 @@ Do not add migration status tables, per-legacy-script checklists, temporary
 coverage counts, or owner queues to this file. Put those in the issue or PR
 that owns the work instead.
 
-## Why
+## What to migrate next
+
+When moving behavior from a legacy E2E script into the scenario framework:
+
+1. Identify the relevant audit issue (#4347–#4356).
+2. Add or update the product-facing manifest only when the desired setup or
+   onboarding state changes.
+3. Add typed scenario registry coverage when the workflow matrix needs a new
+   canonical scenario ID.
+4. Add YAML metadata when the shell runner needs to resolve or execute the plan.
+5. Add reusable suite or assertion helpers instead of copying entire legacy
+   scripts.
+6. Add framework tests that prevent the typed registry, YAML aliases, workflow
+   routes, manifests, and suites from drifting.
+7. Leave legacy executable scripts in place until #4357 records deletion
+   readiness.
+
+## Useful commands
+
+```bash
+# Typed registry inventory and dry-run path
+npx tsx test/e2e-scenario/scenarios/run.ts --list
+npx tsx test/e2e-scenario/scenarios/run.ts --emit-matrix
+npx tsx test/e2e-scenario/scenarios/run.ts --scenarios <id> --plan-only
+npx tsx test/e2e-scenario/scenarios/run.ts --scenarios <id> --dry-run
+
+# YAML/shell resolver and live runner path
+bash test/e2e-scenario/runtime/run-scenario.sh <id> --plan-only
+bash test/e2e-scenario/runtime/run-scenario.sh <id> --dry-run
+bash test/e2e-scenario/runtime/run-scenario.sh <id>
+
+# Metadata and framework validation
+bash test/e2e-scenario/runtime/coverage-report.sh
+npx vitest run --project e2e-scenario-framework --silent=false --reporter=default
+```
+
+## Cleanup rules
 
-The E2E migration moves quickly and is coordinated across multiple issues and
-pull requests. Keeping mutable migration state in GitHub avoids leaving stale
-repo documentation that disagrees with the latest merged code or open review.
+- Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts.
+- Do not reintroduce the removed workflow-level parity report unless maintainers
+  explicitly reopen that direction.
+- Do not delete legacy executable E2Es as part of ordinary domain migration PRs;
+  queue deletion candidates for #4357.
+- Keep docs focused on how the framework works now. Put changing progress status
+  in issues and PRs.

test/e2e-scenario/docs/README.md

@@ -1,109 +1,131 @@
 <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
 <!-- SPDX-License-Identifier: Apache-2.0 -->
 
-# NemoClaw E2E
+# NemoClaw E2E scenario framework
 
-End-to-end tests organized around **setup scenarios** rather than
-one-off shell scripts. A scenario declares *how you got to a working
-NemoClaw* (platform + install + runtime + onboarding); a scenario
-resolves to an **expected state** contract; once that state validates,
-one or more **suites** run functional assertions against it.
+NemoClaw's scenario E2E framework is currently a **hybrid** migration model.
+It combines typed scenario builders, product-facing setup manifests, YAML
+runtime metadata, and reusable shell suites while the older live E2E scripts
+continue to run in parallel.
 
-```text
-setup scenario → expected state → suite sequence
-```
+## Current sources of truth
 
-The durable framework inputs live in these places — read these first,
-they are short and deliberately not redundant with prose:
-
-- [`../nemoclaw_scenarios/scenarios.yaml`](../nemoclaw_scenarios/scenarios.yaml)
-  — YAML metadata for platforms, installs, runtimes, onboarding choices,
-  layered test plans, and legacy scenario aliases.
-- [`../nemoclaw_scenarios/expected-states.yaml`](../nemoclaw_scenarios/expected-states.yaml)
-  — reusable structural contracts (gateway health, sandbox status,
-  inference routing, etc.).
-- [`../validation_suites/suites.yaml`](../validation_suites/suites.yaml)
-  — ordered validation steps, each with a `requires_state` predicate.
-- [`../scenarios/`](../scenarios/)
-  — typed scenario builders, manifests, assertion groups, phase
-  orchestrators, and shared SUT-boundary clients used by scenario
-  framework tests and scenario-workflow planning.
+Use the source that matches the task:
 
-## Layered scenario model
+| Task | Current source |
+| --- | --- |
+| Scenario workflow fan-out and dry-run planning | `test/e2e-scenario/scenarios/registry.ts`, `test/e2e-scenario/scenarios/scenarios/baseline.ts`, and `test/e2e-scenario/scenarios/run.ts` |
+| Product-facing desired setup/onboarding state | `test/e2e-scenario/manifests/*.yaml` |
+| Shell runner scenario resolution and live scenario execution | `test/e2e-scenario/nemoclaw_scenarios/scenarios.yaml`, `expected-states.yaml`, and `validation_suites/suites.yaml` |
+| Reusable live suite assertions | `test/e2e-scenario/validation_suites/` |
+| Existing nightly and platform E2E coverage | legacy `test/e2e/test-*.sh` scripts and their workflows |
 
-The E2E source of truth is layered as base environment, onboarding profile,
-test plan, expected state, and post-onboard suites. Test plans can also declare
-onboarding assertions that run after install/onboard and before expected-state
-validation.
+The migration goal is to keep these surfaces aligned while progressively moving
+coverage into scenario contracts and suites. Do not add new legacy-style
+`test/e2e/test-*.sh` entrypoints unless there is a specific maintainer-approved
+reason.
 
-Plan-only resolution accepts either an alias or a test plan ID:
+## Layered scenario model
 
-```bash
-bash test/e2e-scenario/runtime/run-scenario.sh ubuntu-repo-cloud-openclaw --plan-only
-bash test/e2e-scenario/runtime/run-scenario.sh ubuntu-repo-docker__cloud-nvidia-openclaw --plan-only
+The conceptual model is layered:
+
+```text
+base environment
+  → onboarding profile / manifest
+    → onboarding assertions
+      → expected state
+        → post-onboard suites
 ```
 
+The YAML shell runner expresses this through:
+
+- `base_scenarios`: platform + install + runtime
+- `onboarding_profiles`: user onboarding choices
+- `test_plans`: base + onboarding + expected state + suites
+- `setup_scenarios`: friendly aliases and compatibility metadata
+- `onboarding_assertions`: setup/onboarding checks that run before suites
+
+The typed scenario registry expresses the same intent as deterministic code and
+is used by the scenario workflow matrix and dry-run plan artifacts.
+
 ## How to run
 
 ```bash
-bash test/e2e-scenario/runtime/run-scenario.sh <id> --plan-only       # resolve + print plan, no side effects
-bash test/e2e-scenario/runtime/run-scenario.sh <id> --dry-run         # helpers short-circuit with trace
-bash test/e2e-scenario/runtime/run-scenario.sh <id> --validate-only   # assume setup done; validate expected state
-bash test/e2e-scenario/runtime/run-scenario.sh <id>                   # full live run
-bash test/e2e-scenario/runtime/run-suites.sh <suite-id> [<suite-id>…]
-bash test/e2e-scenario/runtime/coverage-report.sh                     # Markdown matrix of scenario × suite
+# YAML/shell scenario runner
+bash test/e2e-scenario/runtime/run-scenario.sh <id> --plan-only
+bash test/e2e-scenario/runtime/run-scenario.sh <id> --dry-run
+bash test/e2e-scenario/runtime/run-scenario.sh <id> --validate-only
+bash test/e2e-scenario/runtime/run-scenario.sh <id>
+
+# Suite runner against an existing scenario context
+bash test/e2e-scenario/runtime/run-suites.sh <suite-id> [<suite-id>...]
+
+# Scenario metadata coverage report
+bash test/e2e-scenario/runtime/coverage-report.sh
+
+# Typed scenario registry / workflow dry-run path
+npx tsx test/e2e-scenario/scenarios/run.ts --list
+npx tsx test/e2e-scenario/scenarios/run.ts --scenarios <id> --plan-only
+npx tsx test/e2e-scenario/scenarios/run.ts --scenarios <id> --dry-run
+npx tsx test/e2e-scenario/scenarios/run.ts --emit-matrix
 ```
 
-Override the runtime context dir with `E2E_CONTEXT_DIR=<path>` (default
-`.e2e/`, gitignored). The scenario runner and suites communicate only
-through `$E2E_CONTEXT_DIR/context.env` — suites do not rediscover
-setup state.
+Override the runtime context directory with `E2E_CONTEXT_DIR=<path>` (default
+`.e2e/`, gitignored). The shell scenario runner and suites communicate through
+`$E2E_CONTEXT_DIR/context.env`; suites should not rediscover setup state.
 
-## Where things live
+## Repository layout
 
 ```text
 test/e2e-scenario/
-  docs/                              # README.md, MIGRATION.md
-  scenarios/                         # typed builders, assertion groups, phase orchestration
-  manifests/                         # product-facing NemoClawInstance manifests
-  nemoclaw_scenarios/                # YAML scenario inputs + setup machinery
-    scenarios.yaml / expected-states.yaml
-    install/       # install dispatcher + one file per install profile
-    onboard/       # onboard dispatcher + one file per onboarding profile
-    fixtures/      # reusable stubs (fake-openai, fake-{telegram,discord,slack}, older-base-image)
-    helpers/       # scenario-side shell utilities (e.g. emit-context-from-plan.sh)
-  validation_suites/                 # suite definitions and outcome assertions
+  docs/                              # This guide and migration notes
+  manifests/                         # Product-facing NemoClawInstance desired state
+  scenarios/                         # Typed builders, registry, compiler, assertions, dry-run orchestration
+  nemoclaw_scenarios/                # YAML runtime metadata and setup helpers
+    scenarios.yaml
+    expected-states.yaml
+    install/
+    onboard/
+    fixtures/
+    helpers/
+  validation_suites/                 # Suite definitions and shell assertion steps
     suites.yaml
-    sandbox-exec.sh
-    assert/        # outcome assertions (inference, credentials, policy, messaging)
-    smoke/ inference/ hermes/ platform/ security/   # suite scripts grouped by concern
-  runtime/                           # entry points + cross-cutting shared libs
-    run-scenario.sh / run-suites.sh / coverage-report.sh
-    resolver/      # TypeScript: load, plan, validate, coverage (invoked via tsx)
-    lib/           # shared shell helpers: context, env, cleanup, logging, artifacts, sandbox-teardown
+    smoke/
+    inference/
+    messaging/
+    platform/
+    security/
+    sandbox/
+  runtime/                           # Shell runner, suite runner, resolver, coverage report, shared libs
+    run-scenario.sh
+    run-suites.sh
+    coverage-report.sh
+    resolver/
+    lib/
 ```
 
-The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch).
-Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`,
-`wsl-e2e.yaml`, etc.) remain in place during the migration.
-
-Migration status is tracked outside the repository in the parent epic #3588,
-the active audit-coverage issues, and the pull requests that land each batch.
-Do not add per-script migration checklists, temporary coverage counts,
-workflow-level parity reports, or assertion-ledger gates to repo docs; use
-focused code review and the scenario coverage report to decide what to migrate
-next.
-
-## How to add a scenario, state, or suite
-
-Add-a-scenario, add-a-state, and add-a-suite are short edits to the
-three YAML files above, plus shell scripts under
-`nemoclaw_scenarios/install/`, `nemoclaw_scenarios/onboard/`,
-`validation_suites/assert/`, or `validation_suites/<category>/`. The
-schemas in
-[`../runtime/resolver/schema.ts`](../runtime/resolver/schema.ts)
-describe the required shape; `run-scenario.sh <id> --plan-only`
-validates your change without running anything destructive.
+## CI entry points
+
+- `.github/workflows/e2e-scenarios.yaml` runs typed scenario dry-runs for
+  manually selected scenario IDs.
+- `.github/workflows/e2e-scenarios-all.yaml` fans out typed scenario dry-runs
+  from the typed registry matrix.
+- Existing workflows such as `nightly-e2e.yaml`, `e2e-branch-validation.yaml`,
+  `macos-e2e.yaml`, `wsl-e2e.yaml`, `ollama-proxy-e2e.yaml`, and
+  `regression-e2e.yaml` still run legacy live E2E scripts during the migration.
+- `vitest.config.ts` contains the `e2e-scenario-framework` project for framework
+  and metadata tests.
+
+## Migration tracking
+
+Migration status is tracked outside the repository in GitHub issues and PRs,
+not in repo-local checklists. The parent architecture issue is #3588. Active
+audit-coverage work is tracked by the #4347–#4357 issue set, with focused
+follow-ups such as #4378 for specific drift fixes.
+
+The old workflow-level parity report has been removed. Use scenario framework
+tests, the coverage report, PR review, and the audit issues to decide what to
+migrate next.
 
 When adding a suite assertion, emit or preserve a stable `PASS: <id>` /
 `FAIL: <id>` log line, and record migration evidence or follow-up state in the