From 4a4bf4b489931de9bb0d635fc50d37b5ba251a6e Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Sun, 7 Jun 2026 23:26:33 -0700 Subject: [PATCH] docs(e2e): clarify single-runner scenario target Signed-off-by: Carlos Villela --- test/e2e-scenario/docs/MIGRATION.md | 43 ++++++++++++++++++++++---- test/e2e-scenario/docs/README.md | 48 ++++++++++++++++++++++++----- 2 files changed, 78 insertions(+), 13 deletions(-) diff --git a/test/e2e-scenario/docs/MIGRATION.md b/test/e2e-scenario/docs/MIGRATION.md index c5968510b4c..2bbeeafb200 100644 --- a/test/e2e-scenario/docs/MIGRATION.md +++ b/test/e2e-scenario/docs/MIGRATION.md @@ -20,18 +20,43 @@ The scenario E2E migration is in a hybrid phase: - legacy `test/e2e/test-*.sh` scripts still provide most live nightly and platform coverage. +This hybrid shape is not the target end state. #3588 should converge on a +single scenario runner. Until that runner owns live execution, resolver, +assertion, evidence, and redaction behavior, treat YAML and bash runner updates +as bridge work rather than durable architecture. + Do not assume legacy scripts are deletion-ready just because a scenario or suite name exists. The final reconciliation phase must show either evidence-complete coverage or an explicit audit amendment before legacy executable tests are removed. +## Target architecture + +The final scenario framework should have one execution path: + +- typed scenario definitions compile to the runner plan consumed by CI and local + runs; +- the runner owns setup, onboarding, runtime actions, expected-state probes, + assertion execution, expected-failure matching, evidence artifacts, and + secret redaction; +- reusable assertions prefer TypeScript probes and typed clients; +- shell scripts remain only for host, sandbox, process, or platform boundaries + where shell is the thing being tested or the lowest-risk adapter; +- shell execution is wrapped by the runner so environment scoping, timeout, + redaction, artifact capture, and argument validation are consistent. + +When a bridge PR adds behavior to the current YAML/bash runner, preserve the +requirement it proves, but port that requirement into the single-runner path +before removing legacy runner pieces. Do not deepen the bash runner as a second +long-term source of truth. + ## Active issue tracking Use these GitHub issues for status and follow-up work: | Issue | Purpose | | --- | --- | -| #3588 | Parent architecture epic for layered / hybrid scenario E2E | +| #3588 | Parent architecture epic for layered single-runner scenario E2E | | #4347–#4356 | Domain-specific audit-coverage phases | | #4357 | Final audit reconciliation, placeholder cleanup, and deletion-readiness review | | #4378 | Friendly `setup_scenarios` aliases for layered test plans | @@ -61,12 +86,15 @@ When moving behavior from a legacy E2E script into the scenario framework: onboarding state changes. 3. Add typed scenario registry coverage when the workflow matrix needs a new canonical scenario ID. -4. Add YAML metadata when the shell runner needs to resolve or execute the plan. -5. Add reusable suite or assertion helpers instead of copying entire legacy +4. Add or update current YAML metadata only when the existing bridge runner must + keep resolving the scenario during the migration. +5. Add the reusable assertion or probe in the single-runner direction whenever + possible instead of adding new bash-runner-only behavior. +6. Add reusable suite or assertion helpers instead of copying entire legacy scripts. -6. Add framework tests that prevent the typed registry, YAML aliases, workflow - routes, manifests, and suites from drifting. -7. Leave legacy executable scripts in place until #4357 records deletion +7. Add framework tests that prevent the typed registry, YAML aliases, workflow + routes, manifests, suites, and runner behavior from drifting. +8. Leave legacy executable scripts in place until #4357 records deletion readiness. ## Useful commands @@ -91,6 +119,9 @@ npx vitest run --project e2e-scenario-framework --silent=false --reporter=defaul ## Cleanup rules - Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts. +- Prefer single-runner behavior over new YAML/bash-runner behavior; if a bridge + change is unavoidable, document the porting requirement in the owning issue or + PR. - Do not reintroduce the removed workflow-level parity report unless maintainers explicitly reopen that direction. - Do not delete legacy executable E2Es as part of ordinary domain migration PRs; diff --git a/test/e2e-scenario/docs/README.md b/test/e2e-scenario/docs/README.md index c78a9f9d2d6..7f0f1c72463 100644 --- a/test/e2e-scenario/docs/README.md +++ b/test/e2e-scenario/docs/README.md @@ -8,9 +8,15 @@ It combines typed scenario builders, product-facing setup manifests, YAML runtime metadata, and reusable shell suites while the older live E2E scripts continue to run in parallel. +This hybrid model is transitional. The target architecture for #3588 is a +single scenario runner that owns scenario resolution, orchestration, evidence +collection, redaction, and assertion dispatch. Shell scripts should be kept to +the smallest practical set of system-boundary probes or command fixtures, not a +second planning or assertion-control runtime. + ## Current sources of truth -Use the source that matches the task: +Use the source that matches the task while the migration is in progress: | Task | Current source | | --- | --- | @@ -20,10 +26,36 @@ Use the source that matches the task: | Reusable live suite assertions | `test/e2e-scenario/validation_suites/` | | Existing nightly and platform E2E coverage | legacy `test/e2e/test-*.sh` scripts and their workflows | -The migration goal is to keep these surfaces aligned while progressively moving -coverage into scenario contracts and suites. Do not add new legacy-style -`test/e2e/test-*.sh` entrypoints unless there is a specific maintainer-approved -reason. +The near-term migration goal is to keep these surfaces aligned while coverage is +being moved into scenario contracts and suites. The long-term goal is to remove +the split between typed planning and shell execution. Do not add new +legacy-style `test/e2e/test-*.sh` entrypoints unless there is a specific +maintainer-approved reason. + +## Target runner model + +Future scenario coverage should move toward one runner with these properties: + +- the runner compiles one typed plan for each scenario and treats that plan as + the source of truth for setup, onboarding, expected state, suites, assertions, + evidence paths, and expected failures; +- product-facing manifests remain declarative setup inputs, not executable test + programs; +- assertion modules prefer TypeScript probes and typed client helpers; +- shell is used only when the system under test is a shell command, host + process, container command, or platform-specific probe; +- every shell call goes through a controlled spawn boundary with scoped + environment, timeout, redaction, artifact capture, and command/argument + validation; +- bridge work that expands the YAML/bash runner must also identify how that + behavior will move into the single runner before legacy runner paths are + removed. + +The #4347-#4357 audit-phase issues should be read as acceptance coverage +requirements, not as a permanent requirement to keep YAML resolver or bash +runner deliverables. If a phase issue names YAML or shell-runner artifacts, map +that requirement to equivalent single-runner behavior unless maintainers +explicitly decide to keep a bridge path for the current migration step. ## Layered scenario model @@ -37,7 +69,7 @@ base environment → post-onboard suites ``` -The YAML shell runner expresses this through: +The current YAML shell runner expresses this through: - `base_scenarios`: platform + install + runtime - `onboarding_profiles`: user onboarding choices @@ -46,7 +78,9 @@ The YAML shell runner expresses this through: - `onboarding_assertions`: setup/onboarding checks that run before suites The typed scenario registry expresses the same intent as deterministic code and -is used by the scenario workflow matrix and dry-run plan artifacts. +is used by the scenario workflow matrix and dry-run plan artifacts. The target +single runner should collapse these parallel expressions into one executable +plan model. ## How to run