Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions test/e2e-scenario/docs/MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,43 @@ The scenario E2E migration is in a hybrid phase:
- legacy `test/e2e/test-*.sh` scripts still provide most live nightly and
platform coverage.

This hybrid shape is not the target end state. #3588 should converge on a
single scenario runner. Until that runner owns live execution, resolver,
assertion, evidence, and redaction behavior, treat YAML and bash runner updates
as bridge work rather than durable architecture.

Do not assume legacy scripts are deletion-ready just because a scenario or suite
name exists. The final reconciliation phase must show either evidence-complete
coverage or an explicit audit amendment before legacy executable tests are
removed.

## Target architecture

The final scenario framework should have one execution path:

- typed scenario definitions compile to the runner plan consumed by CI and local
runs;
- the runner owns setup, onboarding, runtime actions, expected-state probes,
assertion execution, expected-failure matching, evidence artifacts, and
secret redaction;
- reusable assertions prefer TypeScript probes and typed clients;
- shell scripts remain only for host, sandbox, process, or platform boundaries
where shell is the thing being tested or the lowest-risk adapter;
- shell execution is wrapped by the runner so environment scoping, timeout,
redaction, artifact capture, and argument validation are consistent.

When a bridge PR adds behavior to the current YAML/bash runner, preserve the
requirement it proves, but port that requirement into the single-runner path
before removing legacy runner pieces. Do not deepen the bash runner as a second
long-term source of truth.

## Active issue tracking

Use these GitHub issues for status and follow-up work:

| Issue | Purpose |
| --- | --- |
| #3588 | Parent architecture epic for layered / hybrid scenario E2E |
| #3588 | Parent architecture epic for layered single-runner scenario E2E |
| #4347–#4356 | Domain-specific audit-coverage phases |
| #4357 | Final audit reconciliation, placeholder cleanup, and deletion-readiness review |
| #4378 | Friendly `setup_scenarios` aliases for layered test plans |
Expand Down Expand Up @@ -61,12 +86,15 @@ When moving behavior from a legacy E2E script into the scenario framework:
onboarding state changes.
3. Add typed scenario registry coverage when the workflow matrix needs a new
canonical scenario ID.
4. Add YAML metadata when the shell runner needs to resolve or execute the plan.
5. Add reusable suite or assertion helpers instead of copying entire legacy
4. Add or update current YAML metadata only when the existing bridge runner must
keep resolving the scenario during the migration.
5. Add the reusable assertion or probe in the single-runner direction whenever
possible instead of adding new bash-runner-only behavior.
6. Add reusable suite or assertion helpers instead of copying entire legacy
scripts.
6. Add framework tests that prevent the typed registry, YAML aliases, workflow
routes, manifests, and suites from drifting.
7. Leave legacy executable scripts in place until #4357 records deletion
7. Add framework tests that prevent the typed registry, YAML aliases, workflow
routes, manifests, suites, and runner behavior from drifting.
8. Leave legacy executable scripts in place until #4357 records deletion
readiness.

## Useful commands
Expand All @@ -91,6 +119,9 @@ npx vitest run --project e2e-scenario-framework --silent=false --reporter=defaul
## Cleanup rules

- Prefer new scenario-matrix coverage over new legacy-style `test-*.sh` scripts.
- Prefer single-runner behavior over new YAML/bash-runner behavior; if a bridge
change is unavoidable, document the porting requirement in the owning issue or
PR.
- Do not reintroduce the removed workflow-level parity report unless maintainers
explicitly reopen that direction.
- Do not delete legacy executable E2Es as part of ordinary domain migration PRs;
Expand Down
48 changes: 41 additions & 7 deletions test/e2e-scenario/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,15 @@ It combines typed scenario builders, product-facing setup manifests, YAML
runtime metadata, and reusable shell suites while the older live E2E scripts
continue to run in parallel.

This hybrid model is transitional. The target architecture for #3588 is a
single scenario runner that owns scenario resolution, orchestration, evidence
collection, redaction, and assertion dispatch. Shell scripts should be kept to
the smallest practical set of system-boundary probes or command fixtures, not a
second planning or assertion-control runtime.

## Current sources of truth

Use the source that matches the task:
Use the source that matches the task while the migration is in progress:

| Task | Current source |
| --- | --- |
Expand All @@ -20,10 +26,36 @@ Use the source that matches the task:
| Reusable live suite assertions | `test/e2e-scenario/validation_suites/` |
| Existing nightly and platform E2E coverage | legacy `test/e2e/test-*.sh` scripts and their workflows |

The migration goal is to keep these surfaces aligned while progressively moving
coverage into scenario contracts and suites. Do not add new legacy-style
`test/e2e/test-*.sh` entrypoints unless there is a specific maintainer-approved
reason.
The near-term migration goal is to keep these surfaces aligned while coverage is
being moved into scenario contracts and suites. The long-term goal is to remove
the split between typed planning and shell execution. Do not add new
legacy-style `test/e2e/test-*.sh` entrypoints unless there is a specific
maintainer-approved reason.

## Target runner model

Future scenario coverage should move toward one runner with these properties:

- the runner compiles one typed plan for each scenario and treats that plan as
the source of truth for setup, onboarding, expected state, suites, assertions,
evidence paths, and expected failures;
- product-facing manifests remain declarative setup inputs, not executable test
programs;
- assertion modules prefer TypeScript probes and typed client helpers;
- shell is used only when the system under test is a shell command, host
process, container command, or platform-specific probe;
- every shell call goes through a controlled spawn boundary with scoped
environment, timeout, redaction, artifact capture, and command/argument
validation;
- bridge work that expands the YAML/bash runner must also identify how that
behavior will move into the single runner before legacy runner paths are
removed.

The #4347-#4357 audit-phase issues should be read as acceptance coverage
requirements, not as a permanent requirement to keep YAML resolver or bash
runner deliverables. If a phase issue names YAML or shell-runner artifacts, map
that requirement to equivalent single-runner behavior unless maintainers
explicitly decide to keep a bridge path for the current migration step.

## Layered scenario model

Expand All @@ -37,7 +69,7 @@ base environment
→ post-onboard suites
```

The YAML shell runner expresses this through:
The current YAML shell runner expresses this through:

- `base_scenarios`: platform + install + runtime
- `onboarding_profiles`: user onboarding choices
Expand All @@ -46,7 +78,9 @@ The YAML shell runner expresses this through:
- `onboarding_assertions`: setup/onboarding checks that run before suites

The typed scenario registry expresses the same intent as deterministic code and
is used by the scenario workflow matrix and dry-run plan artifacts.
is used by the scenario workflow matrix and dry-run plan artifacts. The target
single runner should collapse these parallel expressions into one executable
plan model.

## How to run

Expand Down
Loading