refactor(e2e): make onboarding config the setup source - #4270
Closed
jyaunches wants to merge 83 commits into
Closed
Conversation
…ario-model # Conflicts: # test/e2e/nemoclaw_scenarios/scenarios.yaml # test/e2e/runtime/run-scenario.sh
This reverts commit ed6ddde.
Contributor
PR Review AdvisorFindings: 2 needs attention, 5 worth checking, 0 nice ideas Review findings🛠️ Needs attention
🔎 Worth checking
🌱 Nice ideas
Since last review detailsCurrent findings:
This is an automated advisory review. A human maintainer must make the final merge decision. |
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Auto-dispatched E2E: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
E2E Scenario Advisor RecommendationRequired scenario E2E: Dispatch required scenario E2E:
Full scenario advisor summaryE2E Scenario AdvisorBase: Required scenario E2E
Optional scenario E2E
Relevant changed files
|
Contributor
Selective E2E Results — ❌ Some jobs failedRun: 26479992678
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/e2e/runtime/resolver/expected-failure.ts (1)
101-123:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winTreat empty
message_patternas configured, not absent.
load.tsacceptsmessage_pattern: "", but this branch skips the check because it relies on truthiness. That silently broadens the expected-failure contract instead of validating what was authored.Suggested fix
- if (expected.message_pattern) { + if (expected.message_pattern !== undefined) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/runtime/resolver/expected-failure.ts` around lines 101 - 123, The check currently treats an empty string as absent because it uses truthiness; change the condition so an explicit empty message_pattern is treated as configured (e.g., test for undefined/null or presence of the property) so the branch always runs when the author provided message_pattern, including "". Update the conditional around expected.message_pattern in the block that calls compileMessagePattern so it uses something like expected.message_pattern !== undefined (or 'message_pattern' in expected) and keep the existing error handling (compileMessagePattern, pushing to checks, and calling finalize) unchanged.
🧹 Nitpick comments (1)
test/e2e/runtime/resolver/load.ts (1)
7-16: ⚡ Quick winKeep the expected-failure contract in one module.
expected-failure.tsnow defines its own phases/error classes/interface, but this loader still validates againstschema.ts. That leaves two sources of truth for the same YAML contract, so validation and matching can drift independently.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/runtime/resolver/load.ts` around lines 7 - 16, The loader is importing EXPECTED_FAILURE_ERROR_CLASSES, EXPECTED_FAILURE_PHASES, EXPECTED_FAILURE_SIDE_EFFECTS and the ExpectedFailure types from schema.ts while expected-failure.ts now defines its own contract; consolidate the contract by importing these constants and types from the single canonical module (expected-failure.ts) instead of schema.ts (or move the canonical definitions into schema.ts and update expected-failure.ts accordingly), and update the imports in load.ts to reference the chosen canonical symbols (EXPECTED_FAILURE_PHASES, EXPECTED_FAILURE_ERROR_CLASSES, EXPECTED_FAILURE_SIDE_EFFECTS, ExpectedFailure, ExpectedFailurePhase, ExpectedFailureErrorClass, ExpectedFailureSideEffect) so validation and matching use one source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/e2e/lint-conventions.ts`:
- Around line 107-111: The function lintTopLevelLegacyEntrypoints currently
returns an empty array unconditionally which disables the rule; update
lintTopLevelLegacyEntrypoints to enforce the rule by keeping an allowlist of
known compatibility wrappers (e.g., specific filenames) and scanning the
top-level E2E entrypoints for any files matching the blocked pattern (like
test/e2e/test-*.sh or other configured top-level entrypoint names); for each
matching file not in the allowlist, produce a LintFinding describing the
violation so new top-level wrappers are flagged while existing legacy wrappers
remain permitted.
In `@test/e2e/runtime/resolver/plan.ts`:
- Line 74: mergeExpectedFailure is being given expectedStateId which
misattributes scenario validation errors to the expected-state key; update the
call so the scenario context/ID (the variable representing the scenario—e.g.,
scenarioId or context from the resolver scope) is passed as the third argument
instead of expectedStateId: change
mergeExpectedFailure(expectedStateConfig.expected_failure,
legacy.expected_failure, expectedStateId) to
mergeExpectedFailure(expectedStateConfig.expected_failure,
legacy.expected_failure, <scenario context variable>) so errors are formatted as
setup_scenarios.<context>.
---
Outside diff comments:
In `@test/e2e/runtime/resolver/expected-failure.ts`:
- Around line 101-123: The check currently treats an empty string as absent
because it uses truthiness; change the condition so an explicit empty
message_pattern is treated as configured (e.g., test for undefined/null or
presence of the property) so the branch always runs when the author provided
message_pattern, including "". Update the conditional around
expected.message_pattern in the block that calls compileMessagePattern so it
uses something like expected.message_pattern !== undefined (or 'message_pattern'
in expected) and keep the existing error handling (compileMessagePattern,
pushing to checks, and calling finalize) unchanged.
---
Nitpick comments:
In `@test/e2e/runtime/resolver/load.ts`:
- Around line 7-16: The loader is importing EXPECTED_FAILURE_ERROR_CLASSES,
EXPECTED_FAILURE_PHASES, EXPECTED_FAILURE_SIDE_EFFECTS and the ExpectedFailure
types from schema.ts while expected-failure.ts now defines its own contract;
consolidate the contract by importing these constants and types from the single
canonical module (expected-failure.ts) instead of schema.ts (or move the
canonical definitions into schema.ts and update expected-failure.ts
accordingly), and update the imports in load.ts to reference the chosen
canonical symbols (EXPECTED_FAILURE_PHASES, EXPECTED_FAILURE_ERROR_CLASSES,
EXPECTED_FAILURE_SIDE_EFFECTS, ExpectedFailure, ExpectedFailurePhase,
ExpectedFailureErrorClass, ExpectedFailureSideEffect) so validation and matching
use one source of truth.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1f9241b7-013d-427c-be24-71bf023a12fc
📒 Files selected for processing (17)
scripts/e2e/lint-conventions.tstest/e2e-scenario-advisor.test.tstest/e2e/manifests/openclaw-nvidia-custom-policies.yamltest/e2e/manifests/openclaw-nvidia-gateway-port-conflict.yamltest/e2e/manifests/openclaw-nvidia-invalid-key.yamltest/e2e/runtime/resolver/expected-failure.tstest/e2e/runtime/resolver/load.tstest/e2e/runtime/resolver/plan.tstest/e2e/runtime/resolver/schema.tstest/e2e/runtime/resolver/validator.tstest/e2e/scenario-framework-tests/e2e-expected-failure.test.tstest/e2e/scenario-framework-tests/e2e-manifests.test.tstest/e2e/scenario-framework-tests/e2e-scenario-resolver.test.tstest/e2e/scenarios/assertions/registry.tstest/e2e/scenarios/migration-inventory.tstest/e2e/scenarios/scenarios/baseline.tstools/e2e-advisor/scenarios.mts
🚧 Files skipped from review as they are similar to previous changes (4)
- test/e2e/scenario-framework-tests/e2e-manifests.test.ts
- test/e2e/runtime/resolver/validator.ts
- test/e2e/scenario-framework-tests/e2e-scenario-resolver.test.ts
- test/e2e/scenarios/assertions/registry.ts
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.pre-commit-config.yaml (1)
296-302:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPermanently bypassing the source-shape budget defeats its ratcheting purpose.
Setting
NEMOCLAW_SOURCE_SHAPE_ALLOW=1makes the budget check always pass (perfind-source-shape-tests.ts, this setsallowed = POSITIVE_INFINITY). If this bypass is permanent, the pre-commit hook will never enforce the budget, undermining the ratchet mechanism.If this is a temporary migration measure, consider adding a TODO comment or tracking issue. If the budget check is obsolete, remove the hook entirely rather than silently disabling it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.pre-commit-config.yaml around lines 296 - 302, The pre-commit hook "source-shape-test-budget" is permanently bypassed by setting NEMOCLAW_SOURCE_SHAPE_ALLOW=1 in the entry; remove that env override from the entry so the hook actually enforces the budget (modify the block with id: source-shape-test-budget and entry: env NEMOCLAW_SOURCE_SHAPE_ALLOW=1 npm run source-shape:check), or if this was a temporary migration add a one-line TODO with a tracking issue/expiry next to that entry explaining why it was relaxed, or if the check is truly obsolete delete the entire hook block instead of silently disabling it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.pre-commit-config.yaml:
- Around line 296-302: The pre-commit hook "source-shape-test-budget" is
permanently bypassed by setting NEMOCLAW_SOURCE_SHAPE_ALLOW=1 in the entry;
remove that env override from the entry so the hook actually enforces the budget
(modify the block with id: source-shape-test-budget and entry: env
NEMOCLAW_SOURCE_SHAPE_ALLOW=1 npm run source-shape:check), or if this was a
temporary migration add a one-line TODO with a tracking issue/expiry next to
that entry explaining why it was relaxed, or if the check is truly obsolete
delete the entire hook block instead of silently disabling it.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f53b7852-08e1-4bb2-a641-4ab146ee2bcc
📒 Files selected for processing (5)
.pre-commit-config.yamlscripts/e2e/lint-conventions.tsscripts/find-source-shape-tests.tstest/e2e/scenario-framework-tests/e2e-migration-inventory-lock.test.tstest/e2e/scenario-framework-tests/e2e-yaml-source-retirement.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- test/e2e/scenario-framework-tests/e2e-yaml-source-retirement.test.ts
Contributor
Selective E2E Results — ❌ Some jobs failedRun: 26480380781
|
jyaunches
changed the title
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 26483831722
|
Contributor
Selective E2E Results — ❌ Some jobs failedRun: 26483995396
|
Contributor
Selective E2E Results — ❌ Some jobs failedRun: 26484576300
|
12 tasks
Contributor
Author
|
Superseded by #4283, which reapplies this work as a single clean commit without spec/generated artifacts. |
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 26487276419
|
cv
pushed a commit
that referenced
this pull request
May 27, 2026
## Summary This PR reapplies PR #4270 as a single clean commit that isolates the hybrid scenario E2E framework under `test/e2e-scenario`. It preserves the original legacy shell E2E suite under `test/e2e` and removes spec-work/generated artifacts from the new PR diff. ## Related Issue Closes #4258 Refs #3588 Supersedes #4270 ## Changes - Move scenario framework docs, framework tests, manifests, fixtures, runtime helpers, resolver code, assertion scripts, and typed scenario code from `test/e2e/**` to `test/e2e-scenario/**`. - Keep legacy/original E2E tests separate under `test/e2e` with no `test/e2e/test-*.sh` changes. - Update `.github/workflows/e2e-scenarios.yaml`, `vitest.config.ts`, scenario advisor tooling, workflow-boundary tooling, and E2E lint conventions to use `test/e2e-scenario`. - Update scenario framework tests so they validate the new suite location instead of stale `test/e2e` paths. - Exclude spec-work artifacts and generated report/image files from the diff; `test/e2e/docs/parity-inventory.generated.json` remains with the legacy E2E docs instead of moving into the new suite. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [ ] `npx prek run --all-files` passes - [ ] `npm test` passes - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Additional verification run locally: - `npx tsx test/e2e-scenario/scenarios/run.ts --list` - `NEMOCLAW_TEST_TIMEOUT=20000 npx vitest run --project e2e-scenario-framework` (19 files / 166 tests passed) - `rg "test/e2e/(validation_suites|onboarding_assertions|nemoclaw_scenarios|runtime|scenario-framework-tests|scenarios|manifests|docs)" -n . --glob '!node_modules' --glob '!test/e2e-scenario/**' --glob '!specs/**'` (no matches) - `git diff --name-only origin/main..HEAD | rg '^(specs/|\.pi/|.*\.png$)|generated'` (no matches) - `git rev-list --count origin/main..HEAD` → `1` --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Run multiple typed E2E scenarios as a comma-separated batch, produce plan artifacts, and support dry-run execution (including WSL). * **Improvements** * Concurrency, runner selection, and artifact naming now honor multi-scenario inputs. * New plan compiler, richer assertion registry, scenario registry, orchestrators, and stronger manifest validation. * **Tests** * Many new and updated E2E tests covering plan generation, orchestrators, registry, manifests, and platform/failure cases. * **Documentation** * Updated how-to examples and runtime paths for the new scenario runner. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/4283?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/4283?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the hybrid scenario E2E architecture as a dedicated sibling suite under
test/e2e-scenario, while preserving the original legacy shell E2E suite undertest/e2e.Epic / Architecture Review
The implementation matches the epic's hybrid model: product-facing onboarding manifests feed typed scenario builders; assertion composition lives in code modules; the compiler emits plan previews/run plans; phase orchestrators own environment, onboarding, and runtime execution; shared clients wrap NemoClaw system boundaries.
Target architecture diagram from epic #3588
%%{init: {"flowchart": {"htmlLabels": true, "nodeSpacing": 70, "rankSpacing": 95, "curve": "basis"}}}%% flowchart LR %% NemoClaw E2E architecture — hybrid scenario builders + onboarding manifests classDef yaml fill:#f8fafc,stroke:#475569,stroke-width:2px,color:#0f172a classDef builder fill:#eef8e8,stroke:#76B900,stroke-width:3px,color:#10220a classDef module fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#102040 classDef orch fill:#f0fdf4,stroke:#16a34a,stroke-width:2px,color:#052e16 classDef client fill:#f5f3ff,stroke:#7c3aed,stroke-width:2px,color:#24103f classDef sut fill:#fff7ed,stroke:#ea580c,stroke-width:2px,color:#431407 classDef state fill:#ecfeff,stroke:#0891b2,stroke-width:2px,color:#083344 classDef output fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#052e16 classDef note fill:#ffffff,stroke:#334155,stroke-width:1.5px,color:#0f172a %% ---------------------------------------------------------------------- %% 1. Inputs %% ---------------------------------------------------------------------- subgraph C1["1. Inputs"] direction TB Manifest["<b>Onboarding configuration YAML</b><br/>Product-facing desired setup, not an E2E scenario<br/><br/>• install/runtime choices<br/>• agent/provider/model route<br/>• policy/messaging/lifecycle<br/>• durable refs for backup/update"]:::yaml Scenarios["<b>Deterministic scenario builders</b><br/>E2E scenarios are typed code<br/><br/>• stable scenario IDs<br/>• environment/onboarding combinations<br/>• matrix rules<br/>• GitHub targeted execution"]:::builder Assertions["<b>Assertion modules</b><br/>Logical reusable groups in code, not YAML<br/><br/>• environment groups<br/>• onboarding groups<br/>• runtime/domain groups<br/>• stable IDs + evidence output"]:::module end %% ---------------------------------------------------------------------- %% 2. Compile / Preview %% ---------------------------------------------------------------------- subgraph C2["2. Compile / Preview"] direction TB Compiler["<b>Plan compiler</b><br/>Combines builder + onboarding YAML<br/><br/>• loads manifest<br/>• resolves selected scenario<br/>• expands assertion groups<br/>• validates phase compatibility"]:::orch Plan["<b>Plan preview / run plan</b><br/>Visible before execution<br/><br/>• setup/onboarding actions<br/>• ordered phases<br/>• expanded assertion list<br/>• selected SUT boundaries"]:::state end %% ---------------------------------------------------------------------- %% 3. Phase-owned execution %% ---------------------------------------------------------------------- subgraph C3["3. Phase-owned Execution"] direction TB Runner["<div style='min-width:760px'><b>E2E runner</b><br/>Coordinates the full run: orders phases, delegates to every phase orchestrator, passes prior phase results forward, aggregates final results</div>"]:::orch subgraph PhaseOrchestrators["Managed phase orchestrators"] direction LR EnvPhase["<b>Environment Orchestrator</b><br/>Runs setup actions<br/>Runs environment assertions<br/>Emits environment.result"]:::orch OnboardPhase["<b>Onboarding Orchestrator</b><br/>Consumes onboarding config from YAML<br/>Runs onboarding setup/decisions<br/>Runs onboarding assertions<br/>Emits onboarding.result"]:::orch RuntimePhase["<b>Runtime Orchestrator</b><br/>Runs runtime actions/suites<br/>Runs runtime assertions<br/>Emits runtime.result"]:::orch end PhaseAccess["<b>Phase act/observe requests</b><br/>All phase access to NemoClaw goes through shared clients"]:::state Runner --> EnvPhase Runner -- "onboarding setup / decisions" --> OnboardPhase Runner --> RuntimePhase EnvPhase --> PhaseAccess OnboardPhase --> PhaseAccess RuntimePhase --> PhaseAccess end %% ---------------------------------------------------------------------- %% 4. Access layer %% ---------------------------------------------------------------------- subgraph C4["4. Access Layer"] direction TB Clients["<b>Shared E2E clients / adapters</b><br/>Framework wrappers around product boundaries<br/><br/>• HostCliClient<br/>• GatewayClient<br/>• SandboxClient<br/>• AgentClient<br/>• ProviderClient<br/>• StateClient<br/><br/><i>Clients expose act/observe primitives;<br/>phases decide workflow and pass/fail meaning.</i>"]:::client end %% ---------------------------------------------------------------------- %% 5. System Under Test %% ---------------------------------------------------------------------- subgraph C5["5. System Under Test"] direction TB Host["<b>Host Control Plane</b><br/>NemoClaw CLI<br/>install/update scripts<br/>local config/state<br/>Docker/image/cache"]:::sut Gateway["<b>OpenShell Gateway</b><br/>process/API<br/>credential store / broker boundary<br/>inference routing<br/>policy/proxy enforcement<br/>sandbox lifecycle API"]:::sut Sandbox["<b>Sandbox Runtime</b><br/>container boundary<br/>workspace mount<br/>env / CA / proxy config<br/>generated agent config<br/>logs/files"]:::sut Agent["<b>Agent Runtime</b><br/>OpenClaw or Hermes<br/>plugins/tools<br/>agent home/config/state<br/>agent behavior surface"]:::sut Providers["<b>Provider / Integration Plane</b><br/>NVIDIA · Ollama · compatible API<br/>Slack · Discord · Telegram<br/>Brave/web/search<br/>managed/brokered gateways"]:::sut Durable["<b>Durable State Boundary</b><br/>backup/update-relevant state<br/>config snapshots<br/>credential metadata, not raw secrets<br/>workspace refs<br/>image/runtime versions"]:::sut Host -- "starts/configures" --> Gateway Gateway -- "creates/manages" --> Sandbox Sandbox -- "runs" --> Agent Agent -- "calls through routing/policy" --> Providers Host -- "contributes state" --> Durable Gateway -- "contributes state" --> Durable Sandbox -- "contributes state" --> Durable Agent -- "contributes state" --> Durable end %% ---------------------------------------------------------------------- %% 6. Outputs %% ---------------------------------------------------------------------- subgraph C6["6. Outputs"] direction TB PhaseResults["<b>Phase results</b><br/>environment.result<br/>onboarding.result<br/>runtime.result"]:::state Result["<b>result.yaml</b><br/>observed outcome<br/>assertion summaries<br/>artifact pointers<br/>failure layer"]:::output Reports["<b>Human reports</b><br/>plan preview<br/>GitHub Step Summary<br/>operator notes"]:::output Backup["<b>Future backup / update workflow</b><br/>onboarding YAML + observed result<br/>state diff<br/>restore / migration / update validation"]:::output PhaseResults --> Result --> Reports Result --> Backup end %% Main flow: keep lines mostly horizontal and non-overlapping. Manifest -- "desired setup/onboarding config" --> Compiler Scenarios -- "selected scenario ID / matrix rule" --> Compiler Assertions -- "assertion groups" --> Compiler Compiler -- "compile" --> Plan Plan -- "execute" --> Runner RuntimePhase -- "runtime.result" --> PhaseResults %% Access flow. PhaseAccess -- "act/observe requests" --> Clients Clients -- "wraps" --> Host Clients -- "wraps" --> Gateway Clients -- "wraps" --> Sandbox Clients -- "wraps" --> Agent Clients -- "wraps" --> Providers Clients -- "wraps" --> Durable %% Onboarding YAML drives onboarding decisions; result YAML supports future backup/update. Durable -- "observed durable state" --> Backup %% Guardrails kept at bottom to avoid crossing the main flow. G1["<b>Architectural Note</b><br/>YAML describes setup/onboarding desired state; it is not the test scenario."]:::note G2["<b>Architectural Note</b><br/>Scenarios and assertion composition are deterministic code."]:::note G3["<b>Architectural Note</b><br/>Phase orchestrators own phase assertions; clients only wrap SUT boundaries."]:::note Manifest -- "clarifies" --> G1 Scenarios -- "clarifies" --> G2 Assertions -- "clarifies" --> G2 Clients -- "clarifies" --> G3Changes
test/e2eintotest/e2e-scenarioso the legacy/original E2E suite remains separate.test/e2e/test-*.sh,test/e2e/lib/**,test/e2e/e2e-cloud-experimental/**, and related original E2E assets.test/e2e-scenario/scenarios/.test/e2e-scenario/manifests/and compile them into plan previews without raw secrets or suite IDs.test/e2e-scenario/validation_suites/**andtest/e2e-scenario/onboarding_assertions/**so the new suite is self-contained.test/e2e-scenario/**..github/workflows/e2e-scenarios.yaml, advisor tooling, workflow-boundary tooling, lint conventions, and Vitest config to targettest/e2e-scenario.test/e2eafter their active references moved totest/e2e-scenario.Epic Alignment Check
test/e2e-scenario/manifests/*.yamltest/e2e-scenario/scenarios/test/e2e-scenario/scenarios/assertions/test/e2e-scenario/validation_suites/**andtest/e2e-scenario/onboarding_assertions/**--plan-onlypreviewtest/e2e-scenario/scenarios/orchestrators/test/e2e-scenario/scenarios/clients/test/e2e-scenariotest/e2eType of Change
Verification
npx prek run --all-filespassesnpm testpassesmake docsbuilds without warnings (doc changes only)Additional verification run locally:
npx tsx test/e2e-scenario/scenarios/run.ts --listnpx vitest run --project e2e-scenario-frameworkrg "test/e2e/(validation_suites|onboarding_assertions|nemoclaw_scenarios|runtime|scenario-framework-tests|scenarios|manifests|docs)" -n . --glob '!node_modules' --glob '!test/e2e-scenario/**' --glob '!specs/**'Signed-off-by: Julie Yaunches jyaunches@nvidia.com