Skip to content

docs(e2e): choose Vitest fixtures as scenario runner - #4963

Merged
cv merged 1 commit into
mainfrom
codex/e2e-4941-stack-01-direction
Jun 8, 2026
Merged

docs(e2e): choose Vitest fixtures as scenario runner#4963
cv merged 1 commit into
mainfrom
codex/e2e-4941-stack-01-direction

Conversation

@cv

@cv cv commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documents #4941 as a Vitest fixture-based end state for E2E scenarios: one runner, typed NemoClaw scenario fixtures, and shell/YAML only as migration bridges. Draft stack PR 1/7. cc @jyaunches for review.

Related Issue

Refs #4941.

Changes

  • Updates test/e2e-scenario/docs/README.md to describe the final Vitest scenario shape and the smallest expected test.
  • Updates test/e2e-scenario/docs/MIGRATION.md to reconcile the legacy and scenario paths around the single-runner target.
  • Captures the expected direction away from a hybrid shell-script E2E suite.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

Documentation

  • Updated testing infrastructure documentation to clarify the current hybrid approach and target end-state architecture.
  • Refined migration roadmap and priorities to guide future testing framework improvements.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv added the area: e2e End-to-end tests, nightly failures, or validation infrastructure label Jun 8, 2026
@cv cv self-assigned this Jun 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@cv
cv requested a review from jyaunches June 8, 2026 16:41
@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR updates E2E scenario documentation to clarify the target architecture as a single Vitest-based execution path with NemoClaw fixtures owning scenario vocabulary, setup, evidence, cleanup, and assertions, while adjusting migration guidance and CI integration to prioritize fixture-first design.

Changes

E2E Scenario Migration Documentation Alignment

Layer / File(s) Summary
Target single-execution-path architecture
test/e2e-scenario/docs/MIGRATION.md, test/e2e-scenario/docs/README.md
Clarifies that Vitest is the primary execution surface (test discovery, lifecycle, reporting, fixtures, CI), NemoClaw fixtures own scenario vocabulary and helpers (setup, clients, evidence, redaction, cleanup, assertions), and typed scenario/matrix helpers describe stable combinations. Documents the transitional hybrid approach and boundaries for shell script usage (adapter/boundary probes only, wrapped by fixtures).
Fixture-first scenario shape and expression
test/e2e-scenario/docs/README.md, test/e2e-scenario/docs/MIGRATION.md
Expands fixture-first design with Vitest-test-style examples showing how test bodies express product behavior while fixture implementations hide process spawning, artifact paths, cleanup registration, secret gating, and retry classification. Adds issue #4941 to active tracking for fixture-model execution decisions.
Migration prioritization and cleanup rules
test/e2e-scenario/docs/MIGRATION.md, test/e2e-scenario/docs/README.md
Redirects reusable assertions, probes, and bridge behavior explicitly toward Vitest fixtures instead of YAML/bash-runner paths. Adds a transition rule requiring bridging work to identify how behavior will move into Vitest fixtures before legacy runner paths are removed.
CI integration and decision issue tracking
test/e2e-scenario/docs/README.md
Describes live scenario execution target as a separate opt-in Vitest project in CI. Links migration tracking to audit-coverage issues (#4347#4357, #4378) and identifies #4941 as tracking the execution-model decision.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related issues

  • NVIDIA/NemoClaw#4941: The documentation changes explicitly align the migration target to "Vitest + NemoClaw fixtures" and add an active entry for #4941, directly supporting the proposed Vitest-fixtures execution model.

Possibly related PRs

  • NVIDIA/NemoClaw#4939: Both PRs are documentation-only updates to the same files that align on the "single execution path" target architecture and treat YAML/bash runner work as temporary bridge behavior.
  • NVIDIA/NemoClaw#4649: Both PRs update test/e2e-scenario/docs/MIGRATION.md and test/e2e-scenario/docs/README.md to redefine migration tracking and remove repository-local migration checklist language.
  • NVIDIA/NemoClaw#4654: Both PRs update the same E2E scenario documentation to reconcile the migration/cleanup model and align guidance about future work prioritization.

Suggested labels

documentation, area: docs

Suggested reviewers

  • jyaunches
  • prekshivyas

Poem

🐰 Vitest fixtures spring eternal bright,
NemoClaw helpers guide the light,
Bridge behavior soon will fade away,
Single path forward—Vitest's day! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'docs(e2e): choose Vitest fixtures as scenario runner' directly and clearly summarizes the main change: documenting the decision to use Vitest fixtures as the primary E2E scenario execution approach.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-4941-stack-01-direction

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. Docs-only changes under test/e2e-scenario/docs update migration and target architecture guidance. They cannot affect runtime behavior or real assistant user flows, so no E2E jobs are needed.

Optional E2E

  • None.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

  • None. Docs-only changes under test/e2e-scenario/docs; no scenario runtime, metadata, workflow, suite script, onboarding helper, or expected-state behavior is affected.

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/docs/MIGRATION.md
  • test/e2e-scenario/docs/README.md

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@cv
cv marked this pull request as ready for review June 8, 2026 16:46
@cv cv added the v0.0.61 label Jun 8, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-scenario/docs/README.md`:
- Around line 111-114: The provider value in the fixture-first example passed to
openclaw.onboard uses lowercase "nvidia"; change it to the required uppercase
"NVIDIA" (update the call to openclaw.onboard where provider is set) so the
snippet reads provider: "NVIDIA" to comply with the casing guideline.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d9defc73-ed55-4cd7-870c-18176ebaa0d2

📥 Commits

Reviewing files that changed from the base of the PR and between 9a738e0 and 3e98b63.

📒 Files selected for processing (2)
  • test/e2e-scenario/docs/MIGRATION.md
  • test/e2e-scenario/docs/README.md

Comment on lines +111 to +114
const instance = await openclaw.onboard({
agent: "openclaw",
provider: "nvidia",
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix NVIDIA casing in the fixture-first example.

Line 113 uses provider: "nvidia", but Markdown guidance requires NVIDIA in all caps.

Suggested edit
-    provider: "nvidia",
+    provider: "NVIDIA",

As per coding guidelines, "NVIDIA must be all caps (not Nvidia, nvidia)."

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const instance = await openclaw.onboard({
agent: "openclaw",
provider: "nvidia",
});
const instance = await openclaw.onboard({
agent: "openclaw",
provider: "NVIDIA",
});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/docs/README.md` around lines 111 - 114, The provider value
in the fixture-first example passed to openclaw.onboard uses lowercase "nvidia";
change it to the required uppercase "NVIDIA" (update the call to
openclaw.onboard where provider is set) so the snippet reads provider: "NVIDIA"
to comply with the casing guideline.

Source: Coding guidelines

@cv
cv merged commit a0696cb into main Jun 8, 2026
45 checks passed
@cv
cv deleted the codex/e2e-4941-stack-01-direction branch June 8, 2026 18:07
@wscurran wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants