Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 26 additions & 137 deletions test/e2e-scenario/docs/MIGRATION.md
Original file line number Diff line number Diff line change
@@ -1,149 +1,38 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# E2E Migration Tracker
# E2E migration tracking

This PR migrates all existing `test/e2e/test-*.sh` scripts into the
scenario-based runner introduced by PR #3363. Full deep migration
(Strategy B). Legacy scripts remain in the repo during this PR and run
in parallel for 1–2 nightly cycles after merge; a follow-up PR retires
them once parity is verified.
Migration state for the scenario-based E2E framework is intentionally tracked
outside the repository, in GitHub issues and pull requests. This file documents
that policy so the repo does not retain stale per-script checklists, migrated
counts, or ownership tables after the work moves on.

**Merge gate:** All 40 legacy entry points must have a scenario-based
equivalent that produces the same PASS/FAIL outcomes as the legacy
script in a side-by-side CI run.
## Where to track current work

## Reuse being absorbed
- Use issue #3588 as the parent epic for the layered E2E scenario model.
- Use the active audit-coverage phase issues #4347 through #4357 for current
domain-by-domain implementation, review, and reconciliation state.
- Use issue #4378 for the setup-scenario alias cleanup that bridges layered
`test_plans` to friendly scenario IDs.
- Use the pull request for each change as the evidence record for what landed,
what was deferred, and what follow-up issues remain.

Migrating 40 scripts collapses 13 distinct categories of duplication.
Each row maps to a Wave 0 item or an existing helper.
## What belongs in the repo

| # | Category | Fan-in (legacy) | Target absorber | LOC |
|---|---|---|---|---:|
| 1 | Logging helpers (`section` / `info` / `pass` / `fail`) | 28–39 scripts redefine each | `runtime/lib/logging.sh` (Wave 0.B.5) | 1,556 |
| 2 | Non-interactive env exports | 187 inlined lines across 40 scripts | `runtime/lib/env.sh::e2e_env_apply_noninteractive` + convention 0.G.1 | 175 |
| 3 | Repo-root / `SCRIPT_DIR` discovery | 37 lines, 4 competing patterns | One convention (Wave 0.G.2) | 25 |
| 4 | `nemoclaw list` / `status` / gateway state probes | 142 inlined sites | `validation_suites/assert/{gateway,sandbox}-alive.sh` | 500 |
| 5 | `bash install.sh ...` invocations | 24 scripts | `nemoclaw_scenarios/install/dispatch.sh` dispatcher (Wave 0.C.1) | 300 |
| 6 | `nemoclaw onboard ...` variants | 42 invocations, 8+ flag incantations | `nemoclaw_scenarios/onboard/dispatch.sh` + profile handlers | 800 |
| 7 | Docker older-base-image pattern | 3 hand-rolled implementations | `nemoclaw_scenarios/fixtures/older-base-image.sh` (Wave 0.A.1) | 250 |
| 8 | Trap / cleanup / teardown blocks | 112 lines, ~15 patterns | `runtime/lib/cleanup.sh` + convention 0.G.3 | 400 |
| 9 | Fake-endpoint inline setups | 3 inline variants | `nemoclaw_scenarios/fixtures/fake-{openai,telegram,discord,slack}.sh` (Wave 0.A.2–5) | 150 |
| 10 | Sandbox-scoped exec (`nemoclaw shell <sb> -- ...`) | 15 scripts reimplement with drift | `validation_suites/sandbox-exec.sh` (Wave 0.A.6) | 200 |
| 11 | Hermes/OpenClaw pair-variant scripts | 7 paired scripts share ~70% | Shared suite steps; scenario agent via `expected_state.sandbox.agent` | 800 |
| 12 | `section "Phase N: X"` markers | Every script inflates logs with phase text | Step-script filename carries the name (convention 0.G.4) | 300 |
| 13 | Log-capture paths (`/tmp/*.log`) | 25 different conventions; CI artifact upload assumes one | `$E2E_CONTEXT_DIR/logs/` convention 0.G.5 | 300 |
| **Total** | | | | **~5,556** |
Keep durable framework guidance here:

About **25% LOC reduction** net after legacy retirement. The larger win
is drift reduction: when `--yes-i-accept-third-party-software` renames
again, it's a 1-file change instead of a 24-file change.
- how to run the scenario runner,
- where scenario metadata, typed builders, manifests, and suites live,
- how to add or review a scenario, expected state, assertion, or suite,
- stable conventions that should not change with every migration batch.

## Status summary
Do not add migration status tables, per-legacy-script checklists, temporary
coverage counts, or owner queues to this file. Put those in the issue or PR
that owns the work instead.

| Bucket | Legacy LOC | Status |
|---|---:|---|
| Wave 0 — fixtures, asserts, setup splits, conventions, parity workflow | — | ⬜ not started |
| Wave 1 — onboarding baseline | 1,101 | ⬜ |
| Wave 2 — onboarding lifecycle | 2,013 | ⬜ |
| Wave 3 — sandbox lifecycle | 2,891 | ⬜ |
| Wave 4 — rebuild / upgrade | 1,292 | ⬜ |
| Wave 5 — inference variants | 2,593 | ⬜ |
| Wave 6 — Hermes | 1,646 | ⬜ |
| Wave 7 — messaging | 3,397 | ⬜ |
| Wave 8 — security / policy | 2,241 | ⬜ |
| Wave 9 — runtime / platform services | 1,696 | ⬜ |
| Wave 10 — platform + remote | 1,589 | ⬜ |
| Wave 11 — misc | 405 | ⬜ |
| **Total** | **20,864** | **0 / 40 scripts migrated** |
## Why

## Per-script tracker

Legend: ⬜ not started · 🟨 in progress · ✅ migrated · 🔵 parity verified

### Wave 1 — onboarding baseline

- ⬜ `test-full-e2e.sh` (473) → `onboarding/happy-path/` + scenario `ubuntu-curl-cloud-openclaw`
- ⬜ `test-cloud-onboard-e2e.sh` (337) → `onboarding/public-installer/`
- ⬜ `test-cloud-inference-e2e.sh` (291) → extends `inference/cloud/`

### Wave 2 — onboarding lifecycle

- ⬜ `test-double-onboard.sh` (717) → `onboarding/double-onboard/`
- ⬜ `test-gpu-double-onboard.sh` (571) → `onboarding/double-onboard/` on GPU scenario
- ⬜ `test-onboard-repair.sh` (372) → `onboarding/repair/`
- ⬜ `test-onboard-resume.sh` (353) → `onboarding/resume/`

### Wave 3 — sandbox lifecycle

- ⬜ `test-sandbox-operations.sh` (828) → `sandbox/operations/`
- ⬜ `test-sandbox-survival.sh` (721) → `sandbox/survival/`
- ⬜ `test-snapshot-commands.sh` (281) → `sandbox/snapshot/`
- ⬜ `test-diagnostics.sh` (452) → `sandbox/diagnostics/`
- ⬜ `test-issue-2478-crash-loop-recovery.sh` (609) → `sandbox/crash-loop-recovery/`

### Wave 4 — rebuild / upgrade

- ⬜ `test-rebuild-openclaw.sh` (453) → `sandbox/rebuild-openclaw/` (uses `nemoclaw_scenarios/fixtures/older-base-image.sh`)
- ⬜ `test-rebuild-hermes.sh` (401) → `sandbox/rebuild-hermes/`
- ⬜ `test-upgrade-stale-sandbox.sh` (241) → `sandbox/upgrade-stale/`
- ⬜ `test-sandbox-rebuild.sh` (197) → folded into `sandbox/rebuild-openclaw/`

### Wave 5 — inference variants

- ⬜ `test-gpu-e2e.sh` (565) → `inference/ollama-gpu/` (deep port)
- ⬜ `test-ollama-auth-proxy-e2e.sh` (548) → `inference/ollama-auth-proxy/` (deep port)
- ⬜ `test-inference-routing.sh` (715) → `inference/routing-errors/`
- ⬜ `test-kimi-inference-compat.sh` (765) → `inference/kimi-compat/`

### Wave 6 — Hermes

- ⬜ `test-hermes-e2e.sh` (591) → `onboarding/hermes/` (deep port; currently 1-step health)
- ⬜ `test-hermes-slack-e2e.sh` (537) → `messaging/slack/hermes/`
- ⬜ `test-hermes-discord-e2e.sh` (518) → `messaging/discord/hermes/`

### Wave 7 — messaging

- ⬜ `test-messaging-providers.sh` (1,677) → `messaging/providers/{telegram,discord,slack}/`
- ⬜ `test-token-rotation.sh` (575) → `messaging/token-rotation/`
- ⬜ `test-telegram-injection.sh` (475) → `security/telegram-injection/`
- ⬜ `test-messaging-compatible-endpoint.sh` (670) → `messaging/compatible-endpoint/`

### Wave 8 — security / policy

- ⬜ `test-shields-config.sh` (550) → `security/shields/`
- ⬜ `test-network-policy.sh` (579) → `security/network-policy/`
- ⬜ `test-credential-sanitization.sh` (810) → `security/credentials/sanitization/`
- ⬜ `test-credential-migration.sh` (302) → `security/credentials/migration/`

### Wave 9 — runtime / platform services

- ⬜ `test-runtime-overrides.sh` (272) → `sandbox/runtime-overrides/`
- ⬜ `test-overlayfs-autofix.sh` (537) → `sandbox/overlayfs-autofix/`
- ⬜ `test-device-auth-health.sh` (373) → `lifecycle/device-auth-health/`
- ⬜ `test-state-backup-restore.sh` (378) → `lifecycle/state-backup-restore/`
- ⬜ `test-tunnel-lifecycle.sh` (472) → `lifecycle/tunnel-lifecycle/`

### Wave 10 — platform + remote

- ⬜ `test-spark-install.sh` (157) → `platform/spark/`
- ⬜ `test-launchable-smoke.sh` (589) → `platform/launchable/`
- ⬜ `brev-e2e.test.ts` (843) → `platform/brev-remote/`

### Wave 11 — misc

- ⬜ `test-skill-agent-e2e.sh` (244) → `onboarding/skill-agent/`
- ⬜ `test-docs-validation.sh` (161) → `lifecycle/docs-validation/`

## Migration tracking

The old workflow-level parity report has been removed. Migration is tracked by
coverage domain under issue #3588 and its child issues. For each domain, add the
missing primitive layer first, then migrate assertions into scenario plans and
post-onboard suites with stable assertion IDs.

Use the scenario coverage report plus code review to answer:

- which legacy/nightly behaviors are now represented in scenarios,
- which behaviors remain outstanding for the domain issue, and
- which legacy behaviors should be retired rather than ported.
The E2E migration moves quickly and is coordinated across multiple issues and
pull requests. Keeping mutable migration state in GitHub avoids leaving stale
repo documentation that disagrees with the latest merged code or open review.
37 changes: 28 additions & 9 deletions test/e2e-scenario/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,21 @@ one or more **suites** run functional assertions against it.
setup scenario → expected state → suite sequence
```

The declarative sources of truth live in three files — read these
first, they are short and deliberately not redundant with prose:
The durable framework inputs live in these places — read these first,
they are short and deliberately not redundant with prose:

- [`../nemoclaw_scenarios/scenarios.yaml`](../nemoclaw_scenarios/scenarios.yaml)
— platforms, installs, runtimes, onboarding choices, and the
concrete scenarios that combine them.
YAML metadata for platforms, installs, runtimes, onboarding choices,
layered test plans, and legacy scenario aliases.
- [`../nemoclaw_scenarios/expected-states.yaml`](../nemoclaw_scenarios/expected-states.yaml)
— reusable structural contracts (gateway health, sandbox status,
inference routing, etc.).
- [`../validation_suites/suites.yaml`](../validation_suites/suites.yaml)
— ordered validation steps, each with a `requires_state` predicate.
- [`../scenarios/`](../scenarios/)
— typed scenario builders, manifests, assertion groups, phase
orchestrators, and shared SUT-boundary clients used by scenario
framework tests and scenario-workflow planning.

## Layered scenario model

Expand Down Expand Up @@ -58,9 +62,11 @@ setup state.
## Where things live

```text
test/e2e/
test/e2e-scenario/
docs/ # README.md, MIGRATION.md
nemoclaw_scenarios/ # declarative scenario inputs + setup machinery
scenarios/ # typed builders, assertion groups, phase orchestration
manifests/ # product-facing NemoClawInstance manifests
nemoclaw_scenarios/ # YAML scenario inputs + setup machinery
scenarios.yaml / expected-states.yaml
install/ # install dispatcher + one file per install profile
onboard/ # onboard dispatcher + one file per onboarding profile
Expand All @@ -77,9 +83,16 @@ test/e2e/
lib/ # shared shell helpers: context, env, cleanup, logging, artifacts, sandbox-teardown
```

The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch). Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`, `wsl-e2e.yaml`, etc.) remain in place during the migration.
The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch).
Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`,
`wsl-e2e.yaml`, etc.) remain in place during the migration.

Migration coverage is tracked through the layered scenario definitions, suite inventory, and the domain migration issues linked from issue #3588. Do not add a workflow-level parity report or assertion-ledger gate; use focused code review and the scenario coverage report to decide what to migrate next.
Migration status is tracked outside the repository in the parent epic #3588,
the active audit-coverage issues, and the pull requests that land each batch.
Do not add per-script migration checklists, temporary coverage counts,
workflow-level parity reports, or assertion-ledger gates to repo docs; use
focused code review and the scenario coverage report to decide what to migrate
next.

## How to add a scenario, state, or suite

Expand All @@ -93,6 +106,12 @@ describe the required shape; `run-scenario.sh <id> --plan-only`
validates your change without running anything destructive.

When adding a suite assertion, emit or preserve a stable `PASS: <id>` /
`FAIL: <id>` log line, and update migration coverage through the scenario coverage report and the domain issues under `#3588`. Sandbox lifecycle assertions should use `validation_suites/lib/sandbox_lifecycle.sh`, consume `$E2E_CONTEXT_DIR/context.env`, and keep destructive snapshot restore checks isolated in the opt-in `snapshot-lifecycle` suite. Platform-specific scenarios such as GPU, macOS, WSL, Brev, or DGX Spark must also list `runner_requirements` in `scenarios.yaml`.
`FAIL: <id>` log line, and record migration evidence or follow-up state in the
owning issue or PR. Sandbox lifecycle assertions should use
`validation_suites/lib/sandbox_lifecycle.sh`, consume
`$E2E_CONTEXT_DIR/context.env`, and keep destructive snapshot restore checks
isolated in the opt-in `snapshot-lifecycle` suite. Platform-specific scenarios
such as GPU, macOS, WSL, Brev, or DGX Spark must also list
`runner_requirements` in `scenarios.yaml`.

Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts.
104 changes: 20 additions & 84 deletions test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,101 +4,37 @@
import { describe, expect, it } from "vitest";
import fs from "node:fs";
import path from "node:path";
import yaml from "js-yaml";

import { assertionRegistry } from "../scenarios/assertions/registry.ts";
import { migrationInventory } from "../scenarios/migration-inventory.ts";
import { listScenarios } from "../scenarios/registry.ts";

const REPO_ROOT = path.resolve(import.meta.dirname, "../../..");
const SCENARIO_SUITE_DIR = path.join(REPO_ROOT, "test/e2e-scenario");
const SCENARIOS_PATH = path.join(SCENARIO_SUITE_DIR, "nemoclaw_scenarios", "scenarios.yaml");
const EXPECTED_STATES_PATH = path.join(SCENARIO_SUITE_DIR, "nemoclaw_scenarios", "expected-states.yaml");
const SUITES_PATH = path.join(SCENARIO_SUITE_DIR, "validation_suites", "suites.yaml");

type AnyRecord = Record<string, unknown>;

function loadYaml(filePath: string): AnyRecord {
const doc = yaml.load(fs.readFileSync(filePath, "utf8"));
if (!doc || typeof doc !== "object") {
throw new Error(`${filePath} did not parse to an object`);
}
return doc as AnyRecord;
}
const MIGRATION_DOC = path.join(SCENARIO_SUITE_DIR, "docs", "MIGRATION.md");
const README_DOC = path.join(SCENARIO_SUITE_DIR, "docs", "README.md");
const MIGRATION_INVENTORY = path.join(SCENARIO_SUITE_DIR, "scenarios", "migration-inventory.ts");

function keysFrom(record: unknown): string[] {
if (!record || typeof record !== "object" || Array.isArray(record)) {
return [];
}
return Object.keys(record as AnyRecord).sort();
function read(filePath: string): string {
return fs.readFileSync(filePath, "utf8");
}

function expectCovered(kind: keyof typeof migrationInventory, ids: string[]) {
const mappedIds = new Set(migrationInventory[kind].map((entry) => entry.id));
const missing = ids.filter((id) => !mappedIds.has(id));
expect(missing, `missing ${kind} migration target(s): ${missing.join(", ")}`).toEqual([]);
}

describe("hybrid scenario migration inventory lock", () => {
it("old_scenarios_yaml_should_be_non_runtime_reference_only", () => {
const scenarios = loadYaml(SCENARIOS_PATH);

expect(scenarios).toHaveProperty("setup_scenarios");
expect(scenarios).toHaveProperty("base_scenarios");
expect(scenarios).toHaveProperty("onboarding_profiles");
expect(scenarios).toHaveProperty("test_plans");
expect(scenarios).toHaveProperty("onboarding_assertions");
});

it("typed_registry_should_cover_inventory_targets", () => {
const scenarioIds = new Set(listScenarios().map((scenario) => scenario.id));
const missingScenarios = migrationInventory.setupScenarios
.map((entry) => entry.newOwner.replace(/^scenario:/, ""))
.filter((owner) => !scenarioIds.has(owner));

expect(missingScenarios, `missing scenario owners: ${missingScenarios.join(", ")}`).toEqual([]);
});

it("should_fail_when_old_expected_state_missing_new_owner_or_removal_rationale", () => {
const states = loadYaml(EXPECTED_STATES_PATH);
expect(states).toHaveProperty("expected_states");
const expectedStateIds = keysFrom(states.expected_states);
expect(expectedStateIds.length).toBeGreaterThan(0);

expectCovered("expectedStates", expectedStateIds);
});

it("test_should_fail_when_old_validation_suite_script_missing_new_owner_or_removal_rationale", () => {
const suitesDoc = loadYaml(SUITES_PATH);
expect(suitesDoc).toHaveProperty("suites");
const suites = suitesDoc.suites as Record<string, { steps?: Array<{ script?: string }> }>;
const suiteIds = keysFrom(suites);
expect(suiteIds.length).toBeGreaterThan(0);
const scriptIds = Array.from(
new Set(
Object.values(suites)
.flatMap((suite) => suite.steps ?? [])
.map((step) => step.script)
.filter((script): script is string => Boolean(script)),
),
).sort();
const assertionSuiteIds = new Set(assertionRegistry.groups.map((group) => group.suiteId).filter((suiteId): suiteId is string => Boolean(suiteId)));
const missingAssertionGroups = suiteIds.filter((suiteId) => !assertionSuiteIds.has(suiteId));

expectCovered("validationSuites", suiteIds);
expectCovered("validationSuiteScripts", scriptIds);
expect(missingAssertionGroups, `missing assertion groups: ${missingAssertionGroups.join(", ")}`).toEqual([]);
describe("E2E migration tracking hygiene", () => {
it("keeps mutable migration status out of the scenario source tree", () => {
expect(fs.existsSync(MIGRATION_INVENTORY)).toBe(false);
});

it("should_keep_migration_inventory_out_of_runtime_entrypoint", () => {
const runSource = fs.readFileSync(path.join(SCENARIO_SUITE_DIR, "scenarios", "run.ts"), "utf8");
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);

expect(runSource).not.toContain("migration-inventory");
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
});
Comment on lines +23 to 30

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Assert the README still points to issues/PRs.

This test only checks that README.md says migration status is tracked "outside the repository". It would still pass if the README lost the issue/PR guidance that this PR is trying to preserve.

Suggested tightening
   it("documents that migration state lives in issues and PRs", () => {
     const migration = read(MIGRATION_DOC);
     const readme = read(README_DOC);

     expect(migration).toMatch(/tracked\s+outside the repository/);
     expect(migration).toContain("GitHub issues and pull requests");
     expect(readme).toMatch(/Migration status is tracked outside the repository/);
+    expect(readme).toMatch(/parent epic/i);
+    expect(readme).toMatch(/pull requests?/i);
   });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);
expect(runSource).not.toContain("migration-inventory");
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
});
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
expect(readme).toMatch(/parent epic/i);
expect(readme).toMatch(/pull requests?/i);
});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts`
around lines 23 - 30, The test "documents that migration state lives in issues
and PRs" currently only asserts README contains a generic phrase; update it to
explicitly assert that README.md still mentions "GitHub issues and pull
requests" (or the exact guidance string used in MIGRATION_DOC) so it fails if
the README loses the issues/PRs guidance — locate the test (it(...) in
e2e-migration-inventory-lock.test.ts), use the same read(README_DOC) call and
add an expect(readme).toContain("GitHub issues and pull requests") (or match the
exact wording used in MIGRATION_DOC) to tighten the assertion.


it("should_have_seed_reliability_inventory", () => {
const reliabilityExamples = assertionRegistry.groups.flatMap((group) => group.steps.map((step) => step.reliability).filter(Boolean));
it("does not reintroduce stale per-script migration checklists", () => {
const contents = `${read(MIGRATION_DOC)}\n${read(README_DOC)}`;

expect(reliabilityExamples.some((entry) => entry?.retry && entry.timeoutSeconds)).toBe(true);
expect(contents).not.toMatch(/\b\d+\s*\/\s*\d+ scripts migrated\b/i);
expect(contents).not.toContain("Full deep migration");
expect(contents).not.toContain("Per-script tracker");
expect(contents).not.toContain("Merge gate:");
});
});
Loading
Loading