diff --git a/test/e2e-scenario/docs/MIGRATION.md b/test/e2e-scenario/docs/MIGRATION.md index cf3d0e056dc..219db5f5e40 100644 --- a/test/e2e-scenario/docs/MIGRATION.md +++ b/test/e2e-scenario/docs/MIGRATION.md @@ -1,149 +1,38 @@ -# E2E Migration Tracker +# E2E migration tracking -This PR migrates all existing `test/e2e/test-*.sh` scripts into the -scenario-based runner introduced by PR #3363. Full deep migration -(Strategy B). Legacy scripts remain in the repo during this PR and run -in parallel for 1–2 nightly cycles after merge; a follow-up PR retires -them once parity is verified. +Migration state for the scenario-based E2E framework is intentionally tracked +outside the repository, in GitHub issues and pull requests. This file documents +that policy so the repo does not retain stale per-script checklists, migrated +counts, or ownership tables after the work moves on. -**Merge gate:** All 40 legacy entry points must have a scenario-based -equivalent that produces the same PASS/FAIL outcomes as the legacy -script in a side-by-side CI run. +## Where to track current work -## Reuse being absorbed +- Use issue #3588 as the parent epic for the layered E2E scenario model. +- Use the active audit-coverage phase issues #4347 through #4357 for current + domain-by-domain implementation, review, and reconciliation state. +- Use issue #4378 for the setup-scenario alias cleanup that bridges layered + `test_plans` to friendly scenario IDs. +- Use the pull request for each change as the evidence record for what landed, + what was deferred, and what follow-up issues remain. -Migrating 40 scripts collapses 13 distinct categories of duplication. -Each row maps to a Wave 0 item or an existing helper. +## What belongs in the repo -| # | Category | Fan-in (legacy) | Target absorber | LOC | -|---|---|---|---|---:| -| 1 | Logging helpers (`section` / `info` / `pass` / `fail`) | 28–39 scripts redefine each | `runtime/lib/logging.sh` (Wave 0.B.5) | 1,556 | -| 2 | Non-interactive env exports | 187 inlined lines across 40 scripts | `runtime/lib/env.sh::e2e_env_apply_noninteractive` + convention 0.G.1 | 175 | -| 3 | Repo-root / `SCRIPT_DIR` discovery | 37 lines, 4 competing patterns | One convention (Wave 0.G.2) | 25 | -| 4 | `nemoclaw list` / `status` / gateway state probes | 142 inlined sites | `validation_suites/assert/{gateway,sandbox}-alive.sh` | 500 | -| 5 | `bash install.sh ...` invocations | 24 scripts | `nemoclaw_scenarios/install/dispatch.sh` dispatcher (Wave 0.C.1) | 300 | -| 6 | `nemoclaw onboard ...` variants | 42 invocations, 8+ flag incantations | `nemoclaw_scenarios/onboard/dispatch.sh` + profile handlers | 800 | -| 7 | Docker older-base-image pattern | 3 hand-rolled implementations | `nemoclaw_scenarios/fixtures/older-base-image.sh` (Wave 0.A.1) | 250 | -| 8 | Trap / cleanup / teardown blocks | 112 lines, ~15 patterns | `runtime/lib/cleanup.sh` + convention 0.G.3 | 400 | -| 9 | Fake-endpoint inline setups | 3 inline variants | `nemoclaw_scenarios/fixtures/fake-{openai,telegram,discord,slack}.sh` (Wave 0.A.2–5) | 150 | -| 10 | Sandbox-scoped exec (`nemoclaw shell -- ...`) | 15 scripts reimplement with drift | `validation_suites/sandbox-exec.sh` (Wave 0.A.6) | 200 | -| 11 | Hermes/OpenClaw pair-variant scripts | 7 paired scripts share ~70% | Shared suite steps; scenario agent via `expected_state.sandbox.agent` | 800 | -| 12 | `section "Phase N: X"` markers | Every script inflates logs with phase text | Step-script filename carries the name (convention 0.G.4) | 300 | -| 13 | Log-capture paths (`/tmp/*.log`) | 25 different conventions; CI artifact upload assumes one | `$E2E_CONTEXT_DIR/logs/` convention 0.G.5 | 300 | -| **Total** | | | | **~5,556** | +Keep durable framework guidance here: -About **25% LOC reduction** net after legacy retirement. The larger win -is drift reduction: when `--yes-i-accept-third-party-software` renames -again, it's a 1-file change instead of a 24-file change. +- how to run the scenario runner, +- where scenario metadata, typed builders, manifests, and suites live, +- how to add or review a scenario, expected state, assertion, or suite, +- stable conventions that should not change with every migration batch. -## Status summary +Do not add migration status tables, per-legacy-script checklists, temporary +coverage counts, or owner queues to this file. Put those in the issue or PR +that owns the work instead. -| Bucket | Legacy LOC | Status | -|---|---:|---| -| Wave 0 — fixtures, asserts, setup splits, conventions, parity workflow | — | ⬜ not started | -| Wave 1 — onboarding baseline | 1,101 | ⬜ | -| Wave 2 — onboarding lifecycle | 2,013 | ⬜ | -| Wave 3 — sandbox lifecycle | 2,891 | ⬜ | -| Wave 4 — rebuild / upgrade | 1,292 | ⬜ | -| Wave 5 — inference variants | 2,593 | ⬜ | -| Wave 6 — Hermes | 1,646 | ⬜ | -| Wave 7 — messaging | 3,397 | ⬜ | -| Wave 8 — security / policy | 2,241 | ⬜ | -| Wave 9 — runtime / platform services | 1,696 | ⬜ | -| Wave 10 — platform + remote | 1,589 | ⬜ | -| Wave 11 — misc | 405 | ⬜ | -| **Total** | **20,864** | **0 / 40 scripts migrated** | +## Why -## Per-script tracker - -Legend: ⬜ not started · 🟨 in progress · ✅ migrated · 🔵 parity verified - -### Wave 1 — onboarding baseline - -- ⬜ `test-full-e2e.sh` (473) → `onboarding/happy-path/` + scenario `ubuntu-curl-cloud-openclaw` -- ⬜ `test-cloud-onboard-e2e.sh` (337) → `onboarding/public-installer/` -- ⬜ `test-cloud-inference-e2e.sh` (291) → extends `inference/cloud/` - -### Wave 2 — onboarding lifecycle - -- ⬜ `test-double-onboard.sh` (717) → `onboarding/double-onboard/` -- ⬜ `test-gpu-double-onboard.sh` (571) → `onboarding/double-onboard/` on GPU scenario -- ⬜ `test-onboard-repair.sh` (372) → `onboarding/repair/` -- ⬜ `test-onboard-resume.sh` (353) → `onboarding/resume/` - -### Wave 3 — sandbox lifecycle - -- ⬜ `test-sandbox-operations.sh` (828) → `sandbox/operations/` -- ⬜ `test-sandbox-survival.sh` (721) → `sandbox/survival/` -- ⬜ `test-snapshot-commands.sh` (281) → `sandbox/snapshot/` -- ⬜ `test-diagnostics.sh` (452) → `sandbox/diagnostics/` -- ⬜ `test-issue-2478-crash-loop-recovery.sh` (609) → `sandbox/crash-loop-recovery/` - -### Wave 4 — rebuild / upgrade - -- ⬜ `test-rebuild-openclaw.sh` (453) → `sandbox/rebuild-openclaw/` (uses `nemoclaw_scenarios/fixtures/older-base-image.sh`) -- ⬜ `test-rebuild-hermes.sh` (401) → `sandbox/rebuild-hermes/` -- ⬜ `test-upgrade-stale-sandbox.sh` (241) → `sandbox/upgrade-stale/` -- ⬜ `test-sandbox-rebuild.sh` (197) → folded into `sandbox/rebuild-openclaw/` - -### Wave 5 — inference variants - -- ⬜ `test-gpu-e2e.sh` (565) → `inference/ollama-gpu/` (deep port) -- ⬜ `test-ollama-auth-proxy-e2e.sh` (548) → `inference/ollama-auth-proxy/` (deep port) -- ⬜ `test-inference-routing.sh` (715) → `inference/routing-errors/` -- ⬜ `test-kimi-inference-compat.sh` (765) → `inference/kimi-compat/` - -### Wave 6 — Hermes - -- ⬜ `test-hermes-e2e.sh` (591) → `onboarding/hermes/` (deep port; currently 1-step health) -- ⬜ `test-hermes-slack-e2e.sh` (537) → `messaging/slack/hermes/` -- ⬜ `test-hermes-discord-e2e.sh` (518) → `messaging/discord/hermes/` - -### Wave 7 — messaging - -- ⬜ `test-messaging-providers.sh` (1,677) → `messaging/providers/{telegram,discord,slack}/` -- ⬜ `test-token-rotation.sh` (575) → `messaging/token-rotation/` -- ⬜ `test-telegram-injection.sh` (475) → `security/telegram-injection/` -- ⬜ `test-messaging-compatible-endpoint.sh` (670) → `messaging/compatible-endpoint/` - -### Wave 8 — security / policy - -- ⬜ `test-shields-config.sh` (550) → `security/shields/` -- ⬜ `test-network-policy.sh` (579) → `security/network-policy/` -- ⬜ `test-credential-sanitization.sh` (810) → `security/credentials/sanitization/` -- ⬜ `test-credential-migration.sh` (302) → `security/credentials/migration/` - -### Wave 9 — runtime / platform services - -- ⬜ `test-runtime-overrides.sh` (272) → `sandbox/runtime-overrides/` -- ⬜ `test-overlayfs-autofix.sh` (537) → `sandbox/overlayfs-autofix/` -- ⬜ `test-device-auth-health.sh` (373) → `lifecycle/device-auth-health/` -- ⬜ `test-state-backup-restore.sh` (378) → `lifecycle/state-backup-restore/` -- ⬜ `test-tunnel-lifecycle.sh` (472) → `lifecycle/tunnel-lifecycle/` - -### Wave 10 — platform + remote - -- ⬜ `test-spark-install.sh` (157) → `platform/spark/` -- ⬜ `test-launchable-smoke.sh` (589) → `platform/launchable/` -- ⬜ `brev-e2e.test.ts` (843) → `platform/brev-remote/` - -### Wave 11 — misc - -- ⬜ `test-skill-agent-e2e.sh` (244) → `onboarding/skill-agent/` -- ⬜ `test-docs-validation.sh` (161) → `lifecycle/docs-validation/` - -## Migration tracking - -The old workflow-level parity report has been removed. Migration is tracked by -coverage domain under issue #3588 and its child issues. For each domain, add the -missing primitive layer first, then migrate assertions into scenario plans and -post-onboard suites with stable assertion IDs. - -Use the scenario coverage report plus code review to answer: - -- which legacy/nightly behaviors are now represented in scenarios, -- which behaviors remain outstanding for the domain issue, and -- which legacy behaviors should be retired rather than ported. +The E2E migration moves quickly and is coordinated across multiple issues and +pull requests. Keeping mutable migration state in GitHub avoids leaving stale +repo documentation that disagrees with the latest merged code or open review. diff --git a/test/e2e-scenario/docs/README.md b/test/e2e-scenario/docs/README.md index 15ad01d88d7..6658480be24 100644 --- a/test/e2e-scenario/docs/README.md +++ b/test/e2e-scenario/docs/README.md @@ -13,17 +13,21 @@ one or more **suites** run functional assertions against it. setup scenario → expected state → suite sequence ``` -The declarative sources of truth live in three files — read these -first, they are short and deliberately not redundant with prose: +The durable framework inputs live in these places — read these first, +they are short and deliberately not redundant with prose: - [`../nemoclaw_scenarios/scenarios.yaml`](../nemoclaw_scenarios/scenarios.yaml) - — platforms, installs, runtimes, onboarding choices, and the - concrete scenarios that combine them. + — YAML metadata for platforms, installs, runtimes, onboarding choices, + layered test plans, and legacy scenario aliases. - [`../nemoclaw_scenarios/expected-states.yaml`](../nemoclaw_scenarios/expected-states.yaml) — reusable structural contracts (gateway health, sandbox status, inference routing, etc.). - [`../validation_suites/suites.yaml`](../validation_suites/suites.yaml) — ordered validation steps, each with a `requires_state` predicate. +- [`../scenarios/`](../scenarios/) + — typed scenario builders, manifests, assertion groups, phase + orchestrators, and shared SUT-boundary clients used by scenario + framework tests and scenario-workflow planning. ## Layered scenario model @@ -58,9 +62,11 @@ setup state. ## Where things live ```text -test/e2e/ +test/e2e-scenario/ docs/ # README.md, MIGRATION.md - nemoclaw_scenarios/ # declarative scenario inputs + setup machinery + scenarios/ # typed builders, assertion groups, phase orchestration + manifests/ # product-facing NemoClawInstance manifests + nemoclaw_scenarios/ # YAML scenario inputs + setup machinery scenarios.yaml / expected-states.yaml install/ # install dispatcher + one file per install profile onboard/ # onboard dispatcher + one file per onboarding profile @@ -77,9 +83,16 @@ test/e2e/ lib/ # shared shell helpers: context, env, cleanup, logging, artifacts, sandbox-teardown ``` -The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch). Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`, `wsl-e2e.yaml`, etc.) remain in place during the migration. +The CI entry point is `.github/workflows/e2e-scenarios.yaml` (manual dispatch). +Existing legacy workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`, +`wsl-e2e.yaml`, etc.) remain in place during the migration. -Migration coverage is tracked through the layered scenario definitions, suite inventory, and the domain migration issues linked from issue #3588. Do not add a workflow-level parity report or assertion-ledger gate; use focused code review and the scenario coverage report to decide what to migrate next. +Migration status is tracked outside the repository in the parent epic #3588, +the active audit-coverage issues, and the pull requests that land each batch. +Do not add per-script migration checklists, temporary coverage counts, +workflow-level parity reports, or assertion-ledger gates to repo docs; use +focused code review and the scenario coverage report to decide what to migrate +next. ## How to add a scenario, state, or suite @@ -93,6 +106,12 @@ describe the required shape; `run-scenario.sh --plan-only` validates your change without running anything destructive. When adding a suite assertion, emit or preserve a stable `PASS: ` / -`FAIL: ` log line, and update migration coverage through the scenario coverage report and the domain issues under `#3588`. Sandbox lifecycle assertions should use `validation_suites/lib/sandbox_lifecycle.sh`, consume `$E2E_CONTEXT_DIR/context.env`, and keep destructive snapshot restore checks isolated in the opt-in `snapshot-lifecycle` suite. Platform-specific scenarios such as GPU, macOS, WSL, Brev, or DGX Spark must also list `runner_requirements` in `scenarios.yaml`. +`FAIL: ` log line, and record migration evidence or follow-up state in the +owning issue or PR. Sandbox lifecycle assertions should use +`validation_suites/lib/sandbox_lifecycle.sh`, consume +`$E2E_CONTEXT_DIR/context.env`, and keep destructive snapshot restore checks +isolated in the opt-in `snapshot-lifecycle` suite. Platform-specific scenarios +such as GPU, macOS, WSL, Brev, or DGX Spark must also list +`runner_requirements` in `scenarios.yaml`. Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts. diff --git a/test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts b/test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts index c3af81dfca3..c051bdd0883 100644 --- a/test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts +++ b/test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts @@ -4,101 +4,37 @@ import { describe, expect, it } from "vitest"; import fs from "node:fs"; import path from "node:path"; -import yaml from "js-yaml"; - -import { assertionRegistry } from "../scenarios/assertions/registry.ts"; -import { migrationInventory } from "../scenarios/migration-inventory.ts"; -import { listScenarios } from "../scenarios/registry.ts"; const REPO_ROOT = path.resolve(import.meta.dirname, "../../.."); const SCENARIO_SUITE_DIR = path.join(REPO_ROOT, "test/e2e-scenario"); -const SCENARIOS_PATH = path.join(SCENARIO_SUITE_DIR, "nemoclaw_scenarios", "scenarios.yaml"); -const EXPECTED_STATES_PATH = path.join(SCENARIO_SUITE_DIR, "nemoclaw_scenarios", "expected-states.yaml"); -const SUITES_PATH = path.join(SCENARIO_SUITE_DIR, "validation_suites", "suites.yaml"); - -type AnyRecord = Record; - -function loadYaml(filePath: string): AnyRecord { - const doc = yaml.load(fs.readFileSync(filePath, "utf8")); - if (!doc || typeof doc !== "object") { - throw new Error(`${filePath} did not parse to an object`); - } - return doc as AnyRecord; -} +const MIGRATION_DOC = path.join(SCENARIO_SUITE_DIR, "docs", "MIGRATION.md"); +const README_DOC = path.join(SCENARIO_SUITE_DIR, "docs", "README.md"); +const MIGRATION_INVENTORY = path.join(SCENARIO_SUITE_DIR, "scenarios", "migration-inventory.ts"); -function keysFrom(record: unknown): string[] { - if (!record || typeof record !== "object" || Array.isArray(record)) { - return []; - } - return Object.keys(record as AnyRecord).sort(); +function read(filePath: string): string { + return fs.readFileSync(filePath, "utf8"); } -function expectCovered(kind: keyof typeof migrationInventory, ids: string[]) { - const mappedIds = new Set(migrationInventory[kind].map((entry) => entry.id)); - const missing = ids.filter((id) => !mappedIds.has(id)); - expect(missing, `missing ${kind} migration target(s): ${missing.join(", ")}`).toEqual([]); -} - -describe("hybrid scenario migration inventory lock", () => { - it("old_scenarios_yaml_should_be_non_runtime_reference_only", () => { - const scenarios = loadYaml(SCENARIOS_PATH); - - expect(scenarios).toHaveProperty("setup_scenarios"); - expect(scenarios).toHaveProperty("base_scenarios"); - expect(scenarios).toHaveProperty("onboarding_profiles"); - expect(scenarios).toHaveProperty("test_plans"); - expect(scenarios).toHaveProperty("onboarding_assertions"); - }); - - it("typed_registry_should_cover_inventory_targets", () => { - const scenarioIds = new Set(listScenarios().map((scenario) => scenario.id)); - const missingScenarios = migrationInventory.setupScenarios - .map((entry) => entry.newOwner.replace(/^scenario:/, "")) - .filter((owner) => !scenarioIds.has(owner)); - - expect(missingScenarios, `missing scenario owners: ${missingScenarios.join(", ")}`).toEqual([]); - }); - - it("should_fail_when_old_expected_state_missing_new_owner_or_removal_rationale", () => { - const states = loadYaml(EXPECTED_STATES_PATH); - expect(states).toHaveProperty("expected_states"); - const expectedStateIds = keysFrom(states.expected_states); - expect(expectedStateIds.length).toBeGreaterThan(0); - - expectCovered("expectedStates", expectedStateIds); - }); - - it("test_should_fail_when_old_validation_suite_script_missing_new_owner_or_removal_rationale", () => { - const suitesDoc = loadYaml(SUITES_PATH); - expect(suitesDoc).toHaveProperty("suites"); - const suites = suitesDoc.suites as Record }>; - const suiteIds = keysFrom(suites); - expect(suiteIds.length).toBeGreaterThan(0); - const scriptIds = Array.from( - new Set( - Object.values(suites) - .flatMap((suite) => suite.steps ?? []) - .map((step) => step.script) - .filter((script): script is string => Boolean(script)), - ), - ).sort(); - const assertionSuiteIds = new Set(assertionRegistry.groups.map((group) => group.suiteId).filter((suiteId): suiteId is string => Boolean(suiteId))); - const missingAssertionGroups = suiteIds.filter((suiteId) => !assertionSuiteIds.has(suiteId)); - - expectCovered("validationSuites", suiteIds); - expectCovered("validationSuiteScripts", scriptIds); - expect(missingAssertionGroups, `missing assertion groups: ${missingAssertionGroups.join(", ")}`).toEqual([]); +describe("E2E migration tracking hygiene", () => { + it("keeps mutable migration status out of the scenario source tree", () => { + expect(fs.existsSync(MIGRATION_INVENTORY)).toBe(false); }); - it("should_keep_migration_inventory_out_of_runtime_entrypoint", () => { - const runSource = fs.readFileSync(path.join(SCENARIO_SUITE_DIR, "scenarios", "run.ts"), "utf8"); + it("documents that migration state lives in issues and PRs", () => { + const migration = read(MIGRATION_DOC); + const readme = read(README_DOC); - expect(runSource).not.toContain("migration-inventory"); + expect(migration).toMatch(/tracked\s+outside the repository/); + expect(migration).toContain("GitHub issues and pull requests"); + expect(readme).toMatch(/Migration status is tracked outside the repository/); }); - it("should_have_seed_reliability_inventory", () => { - const reliabilityExamples = assertionRegistry.groups.flatMap((group) => group.steps.map((step) => step.reliability).filter(Boolean)); + it("does not reintroduce stale per-script migration checklists", () => { + const contents = `${read(MIGRATION_DOC)}\n${read(README_DOC)}`; - expect(reliabilityExamples.some((entry) => entry?.retry && entry.timeoutSeconds)).toBe(true); + expect(contents).not.toMatch(/\b\d+\s*\/\s*\d+ scripts migrated\b/i); + expect(contents).not.toContain("Full deep migration"); + expect(contents).not.toContain("Per-script tracker"); + expect(contents).not.toContain("Merge gate:"); }); }); diff --git a/test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts b/test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts index f4d9df5f30b..32b9fb2abbe 100644 --- a/test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts +++ b/test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts @@ -7,7 +7,6 @@ import path from "node:path"; import { scenario } from "../scenarios/builder.ts"; import { compileRunPlans } from "../scenarios/compiler.ts"; -import { migrationInventory } from "../scenarios/migration-inventory.ts"; import { buildScenarioRegistry, listScenarios } from "../scenarios/registry.ts"; const REPO_ROOT = path.resolve(import.meta.dirname, "../../.."); @@ -22,25 +21,7 @@ function runScenarioCli(args: string[]) { }); } -function scenarioOwnerIds(): string[] { - return Array.from( - new Set( - [...migrationInventory.setupScenarios, ...migrationInventory.testPlans] - .map((entry) => entry.newOwner) - .filter((owner) => owner.startsWith("scenario:")) - .map((owner) => owner.replace(/^scenario:/, "")), - ), - ).sort(); -} - describe("deterministic scenario registry", () => { - it("test_should_register_canonical_scenarios_for_all_required_old_coverage", () => { - const registeredIds = new Set(listScenarios().map((entry) => entry.id)); - const missing = scenarioOwnerIds().filter((id) => !registeredIds.has(id)); - - expect(missing, `missing canonical scenario IDs: ${missing.join(", ")}`).toEqual([]); - }); - it("test_should_reject_duplicate_scenario_ids", () => { const first = scenario("duplicate-id").manifest("test/e2e-scenario/manifests/openclaw-nvidia.yaml").build(); const second = scenario("duplicate-id").manifest("test/e2e-scenario/manifests/hermes-nvidia.yaml").build(); diff --git a/test/e2e-scenario/scenarios/migration-inventory.ts b/test/e2e-scenario/scenarios/migration-inventory.ts deleted file mode 100644 index d79eae7360d..00000000000 --- a/test/e2e-scenario/scenarios/migration-inventory.ts +++ /dev/null @@ -1,181 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -export type MigrationStatus = "targeted" | "remove-with-rationale"; - -export interface MigrationInventoryEntry { - id: string; - newOwner: string; - status: MigrationStatus; - rationale?: string; -} - -const targeted = (id: string, newOwner: string): MigrationInventoryEntry => ({ - id, - newOwner, - status: "targeted", -}); - -export const migrationInventory = { - setupScenarios: [ - targeted("ubuntu-repo-cloud-openclaw", "scenario:ubuntu-repo-cloud-openclaw"), - targeted("ubuntu-repo-cloud-hermes", "scenario:ubuntu-repo-cloud-hermes"), - targeted("gpu-repo-local-ollama-openclaw", "scenario:gpu-repo-local-ollama-openclaw"), - targeted("macos-repo-cloud-openclaw", "scenario:macos-repo-cloud-openclaw"), - targeted("wsl-repo-cloud-openclaw", "scenario:wsl-repo-cloud-openclaw"), - targeted("brev-launchable-cloud-openclaw", "scenario:brev-launchable-cloud-openclaw"), - targeted("ubuntu-no-docker-preflight-negative", "scenario:ubuntu-no-docker-preflight-negative"), - ], - baseScenarios: [ - targeted("ubuntu-repo-docker", "scenario environment helper:ubuntuRepoDocker"), - targeted("gpu-repo-docker-cdi", "scenario environment helper:gpuRepoDockerCdi"), - targeted("macos-repo-docker", "scenario environment helper:macosRepoDocker"), - targeted("wsl-repo-docker", "scenario environment helper:wslRepoDocker"), - targeted("brev-launchable-remote", "scenario environment helper:brevLaunchableRemote"), - targeted("ubuntu-repo-no-docker", "scenario environment helper:ubuntuRepoNoDocker"), - ], - onboardingProfiles: [ - targeted("cloud-nvidia-openclaw", "manifest:openclaw-nvidia"), - targeted("cloud-nvidia-hermes", "manifest:hermes-nvidia"), - targeted("local-ollama-openclaw", "manifest:openclaw-ollama-gpu"), - targeted("openai-compatible-openclaw", "manifest:openclaw-openai-compatible"), - targeted("cloud-nvidia-openclaw-brave", "manifest:openclaw-nvidia-brave"), - targeted("cloud-nvidia-openclaw-telegram", "manifest:openclaw-nvidia-telegram"), - targeted("cloud-nvidia-openclaw-discord", "manifest:openclaw-nvidia-discord"), - targeted("cloud-nvidia-openclaw-slack", "manifest:openclaw-nvidia-slack"), - targeted("cloud-nvidia-hermes-discord", "manifest:hermes-nvidia-discord"), - targeted("cloud-nvidia-hermes-slack", "manifest:hermes-nvidia-slack"), - targeted("cloud-nvidia-openclaw-resume-after-interrupt", "manifest:openclaw-nvidia-resume"), - targeted("cloud-nvidia-openclaw-repair-existing-config", "manifest:openclaw-nvidia-repair"), - targeted("cloud-nvidia-openclaw-double-same-provider", "manifest:openclaw-nvidia-double-same-provider"), - targeted("cloud-nvidia-openclaw-double-provider-switch", "manifest:openclaw-nvidia-double-provider-switch"), - targeted("cloud-nvidia-openclaw-token-rotation", "manifest:openclaw-nvidia-token-rotation"), - ], - testPlans: [ - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw", "scenario:ubuntu-repo-cloud-openclaw"), - targeted("ubuntu-repo-docker__cloud-nvidia-hermes", "scenario:ubuntu-repo-cloud-hermes"), - targeted("gpu-repo-docker-cdi__local-ollama-openclaw", "scenario:gpu-repo-local-ollama-openclaw"), - targeted("macos-repo-docker__cloud-nvidia-openclaw", "scenario:macos-repo-cloud-openclaw"), - targeted("wsl-repo-docker__cloud-nvidia-openclaw", "scenario:wsl-repo-cloud-openclaw"), - targeted("brev-launchable-remote__cloud-nvidia-openclaw", "scenario:brev-launchable-cloud-openclaw"), - targeted("ubuntu-repo-no-docker__cloud-nvidia-openclaw", "scenario:ubuntu-no-docker-preflight-negative"), - targeted("ubuntu-repo-docker__openai-compatible-openclaw", "scenario:ubuntu-repo-openai-compatible-openclaw"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-brave", "scenario:ubuntu-repo-cloud-openclaw-brave"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-telegram", "scenario:ubuntu-repo-cloud-openclaw-telegram"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-discord", "scenario:ubuntu-repo-cloud-openclaw-discord"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-slack", "scenario:ubuntu-repo-cloud-openclaw-slack"), - targeted("ubuntu-repo-docker__cloud-nvidia-hermes-discord", "scenario:ubuntu-repo-cloud-hermes-discord"), - targeted("ubuntu-repo-docker__cloud-nvidia-hermes-slack", "scenario:ubuntu-repo-cloud-hermes-slack"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-resume-after-interrupt", "scenario:ubuntu-repo-cloud-openclaw-resume"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-repair-existing-config", "scenario:ubuntu-repo-cloud-openclaw-repair"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-double-same-provider", "scenario:ubuntu-repo-cloud-openclaw-double-same-provider"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-double-provider-switch", "scenario:ubuntu-repo-cloud-openclaw-double-provider-switch"), - targeted("ubuntu-repo-docker__cloud-nvidia-openclaw-token-rotation", "scenario:ubuntu-repo-cloud-openclaw-token-rotation"), - ], - expectedStates: [ - targeted("cloud-openclaw-ready", "assertion modules:cloudOpenClawReady"), - targeted("macos-cli-ready-docker-optional", "assertion modules:macosCliDockerOptional"), - targeted("cloud-hermes-ready", "assertion modules:cloudHermesReady"), - targeted("local-ollama-openclaw-ready", "assertion modules:localOllamaOpenClawReady"), - targeted("preflight-failure-no-sandbox", "assertion modules:preflightFailureNoSandbox"), - targeted("cloud-openclaw-custom-policies-ready", "assertion modules:cloudOpenClawCustomPoliciesReady"), - targeted("onboarding-failure-invalid-nvidia-key", "assertion modules:onboardingFailureInvalidNvidiaKey"), - targeted("onboarding-failure-gateway-port-conflict", "assertion modules:onboardingFailureGatewayPortConflict"), - ], - onboardingAssertions: [ - targeted("base-installed", "assertion:onboarding.base.cli-installed"), - targeted("preflight-passed", "assertion:onboarding.preflight.passed"), - targeted("preflight-expected-failed", "assertion:onboarding.preflight.expected-failed"), - ], - validationSuites: [ - targeted("smoke", "assertion:runtime.smoke"), - targeted("inference", "assertion:runtime.inference"), - targeted("credentials", "assertion:runtime.credentials"), - targeted("local-ollama-inference", "assertion:runtime.local-ollama-inference"), - targeted("ollama-proxy", "assertion:runtime.ollama-proxy"), - targeted("platform-macos", "assertion:platform.macos"), - targeted("platform-wsl", "assertion:platform.wsl"), - targeted("hermes-specific", "assertion:runtime.hermes-specific"), - targeted("gateway-health", "assertion:runtime.gateway-health"), - targeted("sandbox-shell", "assertion:runtime.sandbox-shell"), - targeted("cloud-inference", "assertion:runtime.cloud-inference"), - targeted("ollama-auth-proxy", "assertion:runtime.ollama-auth-proxy"), - targeted("security-credentials", "assertion:security.credentials"), - targeted("messaging-telegram", "assertion:messaging.telegram"), - targeted("messaging-discord", "assertion:messaging.discord"), - targeted("messaging-slack", "assertion:messaging.slack"), - targeted("security-shields", "assertion:security.shields"), - targeted("inference-routing", "assertion:runtime.inference-routing"), - targeted("sandbox-lifecycle", "assertion:lifecycle.sandbox-lifecycle"), - targeted("sandbox-operations", "assertion:lifecycle.sandbox-operations"), - targeted("snapshot", "assertion:lifecycle.snapshot"), - targeted("rebuild", "assertion:lifecycle.rebuild"), - targeted("upgrade", "assertion:lifecycle.upgrade"), - targeted("diagnostics", "assertion:diagnostics"), - targeted("docs-validation", "assertion:docs-validation"), - targeted("openai-compatible-inference", "assertion:runtime.openai-compatible-inference"), - targeted("inference-switch", "assertion:runtime.inference-switch"), - targeted("kimi-compatibility", "assertion:runtime.kimi-compatibility"), - targeted("messaging-token-rotation", "assertion:messaging.token-rotation"), - targeted("security-policy", "assertion:security.policy"), - targeted("security-injection", "assertion:security.injection"), - targeted("baseline-onboarding", "assertion:baseline.onboarding"), - targeted("model-router", "assertion:runtime.model-router"), - targeted("onboarding-state", "assertion:onboarding.state"), - targeted("snapshot-lifecycle", "assertion:lifecycle.snapshot"), - ], - validationSuiteScripts: [ - targeted("baseline-onboarding/00-cli-and-openshell.sh", "assertion step:baseline.cli-and-openshell"), - targeted("baseline-onboarding/01-sandbox-state.sh", "assertion step:baseline.sandbox-state"), - targeted("baseline-onboarding/02-route-and-smoke.sh", "assertion step:baseline.route-and-smoke"), - targeted("hermes/00-hermes-health.sh", "assertion step:runtime.hermes.health"), - targeted("inference/cloud/00-models-health.sh", "assertion step:runtime.inference.models-health"), - targeted("inference/cloud/01-chat-completion.sh", "assertion step:runtime.inference.chat-completion"), - targeted("inference/cloud/02-inference-local-from-sandbox.sh", "assertion step:runtime.inference.sandbox-local"), - targeted("inference/kimi-compatibility/00-plugin-wiring.sh", "assertion step:runtime.kimi.plugin-wiring"), - targeted("inference/kimi-compatibility/01-kimi-compatible-models-route.sh", "assertion step:runtime.kimi.compatible-models-route"), - targeted("inference/model-router/00-healthy-endpoint.sh", "assertion step:runtime.model-router.healthy-endpoint"), - targeted("inference/model-router/01-provider-routed-completion.sh", "assertion step:runtime.model-router.provider-routed-completion"), - targeted("inference/ollama-auth-proxy/00-proxy-reachable.sh", "assertion step:runtime.ollama-auth-proxy.reachable"), - targeted("inference/ollama-auth-proxy/01-auth-enforcement.sh", "assertion step:runtime.ollama-auth-proxy.auth-enforcement"), - targeted("inference/ollama-gpu/00-ollama-models-health.sh", "assertion step:runtime.ollama.models-health"), - targeted("inference/routing/00-inference-local-chat-completion.sh", "assertion step:runtime.inference.routing-chat"), - targeted("inference/routing/01-provider-route-health.sh", "assertion step:runtime.inference.provider-route-health"), - targeted("inference/switch/00-route-state-updated.sh", "assertion step:runtime.inference.route-state-updated"), - targeted("inference/switch/01-switched-inference-local-chat.sh", "assertion step:runtime.inference.switched-local-chat"), - targeted("inference/ollama-gpu/01-ollama-chat-completion.sh", "assertion step:runtime.ollama.chat-completion"), - targeted("platform/macos/00-macos-smoke.sh", "assertion step:platform.macos.smoke"), - targeted("platform/wsl/00-wsl-smoke.sh", "assertion step:platform.wsl.smoke"), - targeted("onboarding/state/00-registry-provider-model-policies.sh", "assertion step:onboarding.state.registry"), - targeted("onboarding/state/01-session-provider-model-policies.sh", "assertion step:onboarding.state.session"), - targeted("rebuild_upgrade/00-state-preserved.sh", "assertion step:lifecycle.rebuild.state-preserved"), - targeted("rebuild_upgrade/01-agent-version-upgraded.sh", "assertion step:lifecycle.rebuild.agent-version-upgraded"), - targeted("rebuild_upgrade/02-post-rebuild-inference.sh", "assertion step:lifecycle.rebuild.post-rebuild-inference"), - targeted("rebuild_upgrade/03-policy-config-preserved.sh", "assertion step:lifecycle.upgrade.policy-config-preserved"), - targeted("rebuild_upgrade/04-upgrade-survivor-reachable.sh", "assertion step:lifecycle.upgrade.survivor-reachable"), - targeted("sandbox/lifecycle/00-gateway-health.sh", "assertion step:lifecycle.sandbox.gateway-health"), - targeted("sandbox/lifecycle/01-gateway-recovery.sh", "assertion step:lifecycle.sandbox.gateway-recovery"), - targeted("sandbox/operations/00-list-and-status.sh", "assertion step:lifecycle.sandbox.list-and-status"), - targeted("sandbox/operations/01-logs-and-exec.sh", "assertion step:lifecycle.sandbox.logs-and-exec"), - targeted("sandbox/snapshot/00-create-list-restore.sh", "assertion step:lifecycle.snapshot.create-list-restore"), - targeted("security/credentials/00-credentials-present.sh", "assertion step:security.credentials.present"), - targeted("security/credentials/01-no-plaintext-host-store.sh", "assertion step:security.credentials.no-plaintext-host-store"), - targeted("security/injection/00-telegram-message-not-shell-executed.sh", "assertion step:security.injection.blocked"), - targeted("security/policy/00-telegram-preset-applied.sh", "assertion step:security.policy.telegram-preset"), - targeted("security/policy/01-openshell-version-supports-credential-rewrite.sh", "assertion step:security.policy.credential-rewrite"), - targeted("security/shields/00-config-consistent.sh", "assertion step:security.shields.config"), - targeted("smoke/00-cli-available.sh", "assertion step:runtime.smoke.cli-available"), - targeted("smoke/01-gateway-health.sh", "assertion step:runtime.smoke.gateway-health"), - targeted("smoke/02-sandbox-listed.sh", "assertion step:runtime.smoke.sandbox-listed"), - targeted("smoke/03-sandbox-shell.sh", "assertion step:runtime.smoke.sandbox-shell"), - targeted("messaging/common/00-provider-attached.sh", "assertion step:messaging.common.provider-attached"), - targeted("messaging/common/01-placeholder-configured.sh", "assertion step:messaging.common.placeholder-configured"), - targeted("messaging/common/02-no-secret-leak.sh", "assertion step:messaging.common.no-secret-leak"), - targeted("messaging/common/03-bridge-reachable.sh", "assertion step:messaging.common.bridge-reachable"), - targeted("messaging/discord/00-discord-gateway-path.sh", "assertion step:messaging.discord.gateway-path"), - targeted("messaging/slack/00-slack-provider-state.sh", "assertion step:messaging.slack.provider-state"), - targeted("messaging/telegram/00-telegram-injection-safety.sh", "assertion step:messaging.telegram.injection-safety"), - targeted("messaging/telegram/01-telegram-injection-payload-classes.sh", "assertion step:messaging.telegram.injection-payload-classes"), - targeted("messaging/token-rotation/00-provider-rotation-isolated.sh", "assertion step:messaging.token-rotation"), - ], -} as const;