diff --git a/test/e2e-scenario/docs/MIGRATION.md b/test/e2e-scenario/docs/MIGRATION.md index 219db5f5e40..c5968510b4c 100644 --- a/test/e2e-scenario/docs/MIGRATION.md +++ b/test/e2e-scenario/docs/MIGRATION.md @@ -1,22 +1,43 @@ -# E2E migration tracking +# E2E scenario migration notes -Migration state for the scenario-based E2E framework is intentionally tracked -outside the repository, in GitHub issues and pull requests. This file documents -that policy so the repo does not retain stale per-script checklists, migrated -counts, or ownership tables after the work moves on. +This file records the current migration model for contributors. It is not the +source of truth for per-domain status. Mutable migration state is tracked +outside the repository in GitHub issues and pull requests so reviewers can +discuss, update, and close work in one place. -## Where to track current work +## Current migration state -- Use issue #3588 as the parent epic for the layered E2E scenario model. -- Use the active audit-coverage phase issues #4347 through #4357 for current - domain-by-domain implementation, review, and reconciliation state. -- Use issue #4378 for the setup-scenario alias cleanup that bridges layered - `test_plans` to friendly scenario IDs. -- Use the pull request for each change as the evidence record for what landed, - what was deferred, and what follow-up issues remain. +The scenario E2E migration is in a hybrid phase: + +- typed scenario builders drive scenario workflow fan-out and dry-run plans; +- product-facing `NemoClawInstance` manifests describe desired setup and + onboarding state; +- YAML metadata still drives the shell scenario runner and live suite + resolution; +- legacy `test/e2e/test-*.sh` scripts still provide most live nightly and + platform coverage. + +Do not assume legacy scripts are deletion-ready just because a scenario or suite +name exists. The final reconciliation phase must show either evidence-complete +coverage or an explicit audit amendment before legacy executable tests are +removed. + +## Active issue tracking + +Use these GitHub issues for status and follow-up work: + +| Issue | Purpose | +| --- | --- | +| #3588 | Parent architecture epic for layered / hybrid scenario E2E | +| #4347–#4356 | Domain-specific audit-coverage phases | +| #4357 | Final audit reconciliation, placeholder cleanup, and deletion-readiness review | +| #4378 | Friendly `setup_scenarios` aliases for layered test plans | + +If a migration discovery needs durable tracking, add it to the relevant issue or +open a focused child issue. Avoid adding long-lived checklists here. ## What belongs in the repo @@ -31,8 +52,48 @@ Do not add migration status tables, per-legacy-script checklists, temporary coverage counts, or owner queues to this file. Put those in the issue or PR that owns the work instead. -## Why +## What to migrate next + +When moving behavior from a legacy E2E script into the scenario framework: + +1. Identify the relevant audit issue (#4347–#4356). +2. Add or update the product-facing manifest only when the desired setup or + onboarding state changes. +3. Add typed scenario registry coverage when the workflow matrix needs a new + canonical scenario ID. +4. Add YAML metadata when the shell runner needs to resolve or execute the plan. +5. Add reusable suite or assertion helpers instead of copying entire legacy + scripts. +6. Add framework tests that prevent the typed registry, YAML aliases, workflow + routes, manifests, and suites from drifting. +7. Leave legacy executable scripts in place until #4357 records deletion + readiness. + +## Useful commands + +```bash +# Typed registry inventory and dry-run path +npx tsx test/e2e-scenario/scenarios/run.ts --list +npx tsx test/e2e-scenario/scenarios/run.ts --emit-matrix +npx tsx test/e2e-scenario/scenarios/run.ts --scenarios --plan-only +npx tsx test/e2e-scenario/scenarios/run.ts --scenarios --dry-run + +# YAML/shell resolver and live runner path +bash test/e2e-scenario/runtime/run-scenario.sh --plan-only +bash test/e2e-scenario/runtime/run-scenario.sh --dry-run +bash test/e2e-scenario/runtime/run-scenario.sh + +# Metadata and framework validation +bash test/e2e-scenario/runtime/coverage-report.sh +npx vitest run --project e2e-scenario-framework --silent=false --reporter=default +``` + +## Cleanup rules -The E2E migration moves quickly and is coordinated across multiple issues and -pull requests. Keeping mutable migration state in GitHub avoids leaving stale -repo documentation that disagrees with the latest merged code or open review. +- Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts. +- Do not reintroduce the removed workflow-level parity report unless maintainers + explicitly reopen that direction. +- Do not delete legacy executable E2Es as part of ordinary domain migration PRs; + queue deletion candidates for #4357. +- Keep docs focused on how the framework works now. Put changing progress status + in issues and PRs. diff --git a/test/e2e-scenario/docs/README.md b/test/e2e-scenario/docs/README.md index 6658480be24..c78a9f9d2d6 100644 --- a/test/e2e-scenario/docs/README.md +++ b/test/e2e-scenario/docs/README.md @@ -1,109 +1,131 @@ -# NemoClaw E2E +# NemoClaw E2E scenario framework -End-to-end tests organized around **setup scenarios** rather than -one-off shell scripts. A scenario declares *how you got to a working -NemoClaw* (platform + install + runtime + onboarding); a scenario -resolves to an **expected state** contract; once that state validates, -one or more **suites** run functional assertions against it. +NemoClaw's scenario E2E framework is currently a **hybrid** migration model. +It combines typed scenario builders, product-facing setup manifests, YAML +runtime metadata, and reusable shell suites while the older live E2E scripts +continue to run in parallel. -```text -setup scenario → expected state → suite sequence -``` +## Current sources of truth -The durable framework inputs live in these places — read these first, -they are short and deliberately not redundant with prose: - -- [`../nemoclaw_scenarios/scenarios.yaml`](../nemoclaw_scenarios/scenarios.yaml) - — YAML metadata for platforms, installs, runtimes, onboarding choices, - layered test plans, and legacy scenario aliases. -- [`../nemoclaw_scenarios/expected-states.yaml`](../nemoclaw_scenarios/expected-states.yaml) - — reusable structural contracts (gateway health, sandbox status, - inference routing, etc.). -- [`../validation_suites/suites.yaml`](../validation_suites/suites.yaml) - — ordered validation steps, each with a `requires_state` predicate. -- [`../scenarios/`](../scenarios/) - — typed scenario builders, manifests, assertion groups, phase - orchestrators, and shared SUT-boundary clients used by scenario - framework tests and scenario-workflow planning. +Use the source that matches the task: -## Layered scenario model +| Task | Current source | +| --- | --- | +| Scenario workflow fan-out and dry-run planning | `test/e2e-scenario/scenarios/registry.ts`, `test/e2e-scenario/scenarios/scenarios/baseline.ts`, and `test/e2e-scenario/scenarios/run.ts` | +| Product-facing desired setup/onboarding state | `test/e2e-scenario/manifests/*.yaml` | +| Shell runner scenario resolution and live scenario execution | `test/e2e-scenario/nemoclaw_scenarios/scenarios.yaml`, `expected-states.yaml`, and `validation_suites/suites.yaml` | +| Reusable live suite assertions | `test/e2e-scenario/validation_suites/` | +| Existing nightly and platform E2E coverage | legacy `test/e2e/test-*.sh` scripts and their workflows | -The E2E source of truth is layered as base environment, onboarding profile, -test plan, expected state, and post-onboard suites. Test plans can also declare -onboarding assertions that run after install/onboard and before expected-state -validation. +The migration goal is to keep these surfaces aligned while progressively moving +coverage into scenario contracts and suites. Do not add new legacy-style +`test/e2e/test-*.sh` entrypoints unless there is a specific maintainer-approved +reason. -Plan-only resolution accepts either an alias or a test plan ID: +## Layered scenario model -```bash -bash test/e2e-scenario/runtime/run-scenario.sh ubuntu-repo-cloud-openclaw --plan-only -bash test/e2e-scenario/runtime/run-scenario.sh ubuntu-repo-docker__cloud-nvidia-openclaw --plan-only +The conceptual model is layered: + +```text +base environment + → onboarding profile / manifest + → onboarding assertions + → expected state + → post-onboard suites ``` +The YAML shell runner expresses this through: + +- `base_scenarios`: platform + install + runtime +- `onboarding_profiles`: user onboarding choices +- `test_plans`: base + onboarding + expected state + suites +- `setup_scenarios`: friendly aliases and compatibility metadata +- `onboarding_assertions`: setup/onboarding checks that run before suites + +The typed scenario registry expresses the same intent as deterministic code and +is used by the scenario workflow matrix and dry-run plan artifacts. + ## How to run ```bash -bash test/e2e-scenario/runtime/run-scenario.sh --plan-only # resolve + print plan, no side effects -bash test/e2e-scenario/runtime/run-scenario.sh --dry-run # helpers short-circuit with trace -bash test/e2e-scenario/runtime/run-scenario.sh --validate-only # assume setup done; validate expected state -bash test/e2e-scenario/runtime/run-scenario.sh # full live run -bash test/e2e-scenario/runtime/run-suites.sh […] -bash test/e2e-scenario/runtime/coverage-report.sh # Markdown matrix of scenario × suite +# YAML/shell scenario runner +bash test/e2e-scenario/runtime/run-scenario.sh --plan-only +bash test/e2e-scenario/runtime/run-scenario.sh --dry-run +bash test/e2e-scenario/runtime/run-scenario.sh --validate-only +bash test/e2e-scenario/runtime/run-scenario.sh + +# Suite runner against an existing scenario context +bash test/e2e-scenario/runtime/run-suites.sh [...] + +# Scenario metadata coverage report +bash test/e2e-scenario/runtime/coverage-report.sh + +# Typed scenario registry / workflow dry-run path +npx tsx test/e2e-scenario/scenarios/run.ts --list +npx tsx test/e2e-scenario/scenarios/run.ts --scenarios --plan-only +npx tsx test/e2e-scenario/scenarios/run.ts --scenarios --dry-run +npx tsx test/e2e-scenario/scenarios/run.ts --emit-matrix ``` -Override the runtime context dir with `E2E_CONTEXT_DIR=` (default -`.e2e/`, gitignored). The scenario runner and suites communicate only -through `$E2E_CONTEXT_DIR/context.env` — suites do not rediscover -setup state. +Override the runtime context directory with `E2E_CONTEXT_DIR=` (default +`.e2e/`, gitignored). The shell scenario runner and suites communicate through +`$E2E_CONTEXT_DIR/context.env`; suites should not rediscover setup state. -## Where things live +## Repository layout ```text test/e2e-scenario/ - docs/ # README.md, MIGRATION.md - scenarios/ # typed builders, assertion groups, phase orchestration - manifests/ # product-facing NemoClawInstance manifests - nemoclaw_scenarios/ # YAML scenario inputs + setup machinery - scenarios.yaml / expected-states.yaml - install/ # install dispatcher + one file per install profile - onboard/ # onboard dispatcher + one file per onboarding profile - fixtures/ # reusable stubs (fake-openai, fake-{telegram,discord,slack}, older-base-image) - helpers/ # scenario-side shell utilities (e.g. emit-context-from-plan.sh) - validation_suites/ # suite definitions and outcome assertions + docs/ # This guide and migration notes + manifests/ # Product-facing NemoClawInstance desired state + scenarios/ # Typed builders, registry, compiler, assertions, dry-run orchestration + nemoclaw_scenarios/ # YAML runtime metadata and setup helpers + scenarios.yaml + expected-states.yaml + install/ + onboard/ + fixtures/ + helpers/ + validation_suites/ # Suite definitions and shell assertion steps suites.yaml - sandbox-exec.sh - assert/ # outcome assertions (inference, credentials, policy, messaging) - smoke/ inference/ hermes/ platform/ security/ # suite scripts grouped by concern - runtime/ # entry points + cross-cutting shared libs - run-scenario.sh / run-suites.sh / coverage-report.sh - resolver/ # TypeScript: load, plan, validate, coverage (invoked via tsx) - lib/ # shared shell helpers: context, env, cleanup, logging, artifacts, sandbox-teardown + smoke/ + inference/ + messaging/ + platform/ + security/ + sandbox/ + runtime/ # Shell runner, suite runner, resolver, coverage report, shared libs + run-scenario.sh + run-suites.sh + coverage-report.sh + resolver/ + lib/ ``` -The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch). -Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`, -`wsl-e2e.yaml`, etc.) remain in place during the migration. - -Migration status is tracked outside the repository in the parent epic #3588, -the active audit-coverage issues, and the pull requests that land each batch. -Do not add per-script migration checklists, temporary coverage counts, -workflow-level parity reports, or assertion-ledger gates to repo docs; use -focused code review and the scenario coverage report to decide what to migrate -next. - -## How to add a scenario, state, or suite - -Add-a-scenario, add-a-state, and add-a-suite are short edits to the -three YAML files above, plus shell scripts under -`nemoclaw_scenarios/install/`, `nemoclaw_scenarios/onboard/`, -`validation_suites/assert/`, or `validation_suites//`. The -schemas in -[`../runtime/resolver/schema.ts`](../runtime/resolver/schema.ts) -describe the required shape; `run-scenario.sh --plan-only` -validates your change without running anything destructive. +## CI entry points + +- `.github/workflows/e2e-scenarios.yaml` runs typed scenario dry-runs for + manually selected scenario IDs. +- `.github/workflows/e2e-scenarios-all.yaml` fans out typed scenario dry-runs + from the typed registry matrix. +- Existing workflows such as `nightly-e2e.yaml`, `e2e-branch-validation.yaml`, + `macos-e2e.yaml`, `wsl-e2e.yaml`, `ollama-proxy-e2e.yaml`, and + `regression-e2e.yaml` still run legacy live E2E scripts during the migration. +- `vitest.config.ts` contains the `e2e-scenario-framework` project for framework + and metadata tests. + +## Migration tracking + +Migration status is tracked outside the repository in GitHub issues and PRs, +not in repo-local checklists. The parent architecture issue is #3588. Active +audit-coverage work is tracked by the #4347–#4357 issue set, with focused +follow-ups such as #4378 for specific drift fixes. + +The old workflow-level parity report has been removed. Use scenario framework +tests, the coverage report, PR review, and the audit issues to decide what to +migrate next. When adding a suite assertion, emit or preserve a stable `PASS: ` / `FAIL: ` log line, and record migration evidence or follow-up state in the