test(e2e): make docs validation a first-class Vitest lane - #6011
Merged
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a ChangesDocs Validation CI Job and Boundary Validation
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: Dispatch required Vitest E2E scenarios:
Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorBase: Required Vitest E2E scenarios
Optional Vitest E2E scenarios
Relevant changed files
|
Contributor
PR Review Advisor (Nemotron Ultra) — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
🚨 Required before mergeAddress these before merging unless a maintainer explicitly overrides the advisor with rationale.
|
Contributor
PR Review Advisor — No blocking findingsMerge posture: No blocking advisor findings Action checklist
Findings index
Review findings by urgency: 0 required fixes, 0 items to resolve/justify, 1 in-scope improvement
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tools/e2e-scenarios/docs-validation-workflow-boundary.mts (1)
60-65: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid hard-coding the shared selector gate here.
Pinning the entire
if:expression in this per-job validator creates another source of truth for ordinary free-standing job selection. That is likely to drift the next time the shared selector logic changes, even if the workflow remains valid. Based on learnings, "enforce selector/gate invariants for ordinary free-standing jobs centrally—not per job" and "validateFreeStandingInventoryBoundary delegates to validateFreeStandingJobSelector".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/e2e-scenarios/docs-validation-workflow-boundary.mts` around lines 60 - 65, The validator is hard-coding the shared selector gate in the per-job check, which duplicates the source of truth and will drift from the central selector logic. Update validateFreeStandingInventoryBoundary to avoid asserting the full if expression for this job, and rely on validateFreeStandingJobSelector for the shared gate while keeping only the job-specific boundary invariant for JOB_NAME/docs-validation-vitest.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/e2e-scenarios/docs-validation-workflow-boundary.mts`:
- Around line 34-37: Normalize the parsed result in readDocsValidationWorkflow
before it is used as a DocsValidationWorkflow. YAML.parse() may return null or a
non-object, so add a shape check/defaulting step and only return a validated
workflow object; if the file is empty or malformed, surface a controlled
validation error instead of letting later workflow.jobs access throw. Use the
readDocsValidationWorkflow symbol to locate the parsing and conversion logic.
---
Nitpick comments:
In `@tools/e2e-scenarios/docs-validation-workflow-boundary.mts`:
- Around line 60-65: The validator is hard-coding the shared selector gate in
the per-job check, which duplicates the source of truth and will drift from the
central selector logic. Update validateFreeStandingInventoryBoundary to avoid
asserting the full if expression for this job, and rely on
validateFreeStandingJobSelector for the shared gate while keeping only the
job-specific boundary invariant for JOB_NAME/docs-validation-vitest.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ec35f120-33ca-4ede-a296-a7b54a8fb84c
📒 Files selected for processing (3)
.github/workflows/e2e-vitest-scenarios.yamltest/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.tstools/e2e-scenarios/docs-validation-workflow-boundary.mts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
Vitest E2E Scenario Results — ✅ All requested jobs passedRun: 28397028669
|
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.ts (1)
66-73: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the empty-workflow aggregation error too.
This case only checks one boundary error, but an empty workflow also deterministically violates the
report-to-prdependency contract. As written, the test would still pass if that aggregation check regressed. Add the second observable error, or assert the full returned set.Suggested tightening
- expect(validateDocsValidationWorkflowBoundary(workflowPath)).toContain( - "docs-validation-vitest must depend on generate-matrix", - ); + expect(validateDocsValidationWorkflowBoundary(workflowPath)).toEqual( + expect.arrayContaining([ + "docs-validation-vitest must depend on generate-matrix", + "report-to-pr must wait for docs-validation-vitest", + ]), + );As per path instructions,
**/*.test.{ts,js,mts,mjs,cts,cjs}: "Review tests for behavioral confidence rather than implementation lock-in."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.ts` around lines 66 - 73, The empty-workflow test currently only asserts the `docs-validation-vitest must depend on generate-matrix` boundary error, so it can miss a regression in the `report-to-pr` aggregation check. Update `validateDocsValidationWorkflowBoundary` coverage in `docs-validation-workflow-boundary.test.ts` to assert the second deterministic error as well, or verify the full returned error set for the empty workflow so both contract violations are covered.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.ts`:
- Around line 66-73: The empty-workflow test currently only asserts the
`docs-validation-vitest must depend on generate-matrix` boundary error, so it
can miss a regression in the `report-to-pr` aggregation check. Update
`validateDocsValidationWorkflowBoundary` coverage in
`docs-validation-workflow-boundary.test.ts` to assert the second deterministic
error as well, or verify the full returned error set for the empty workflow so
both contract violations are covered.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3eb28ccb-f02a-48f1-bd9a-64ff87a3e5dc
📒 Files selected for processing (2)
test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.tstools/e2e-scenarios/docs-validation-workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (1)
- tools/e2e-scenarios/docs-validation-workflow-boundary.mts
Collaborator
Author
|
Focused runtime verification is green: docs-validation-vitest run 28397028669 completed successfully on head Advisor dispositions:
|
57 tasks
21 tasks
jyaunches
pushed a commit
that referenced
this pull request
Jun 30, 2026
## Summary Refreshes the v0.0.70 release docs from the release announcement and the `v0.0.69..v0.0.70` commit range. It also documents the `channels start` policy restoration behavior that was missing from the shared OpenClaw and Hermes command references, and bumps the Fern CLI version used for docs validation. ## Changes - Replaced the stale `v0.0.70` release-notes entry with the actual release themes, including CLI, onboarding, inference, messaging, Windows, documentation, and release-validation changes. - Documented that `channels start` reapplies the matching built-in network policy preset before rebuild and rolls back to disabled if policy restoration fails. - Bumped `fern/fern.config.json` from `5.55.0` to `5.59.0` for the docs refresh. - Source summary: - #5754 -> `docs/about/release-notes.mdx`: Notes Docker Desktop gateway bridge retry behavior during onboarding. - #5930 -> `docs/about/release-notes.mdx`: Links `nemoclaw use` default sandbox selection to the command reference. - #5948 -> `docs/about/release-notes.mdx`: Links reasoning-compatible endpoint validation to inference documentation. - #5950 -> `docs/about/release-notes.mdx`: Links Windows bootstrap WSL recovery behavior to Windows preparation and troubleshooting docs. - #5856 -> `docs/about/release-notes.mdx`: Notes rebuilt policy preset registry repair. - #5882 and #5949 -> `docs/about/release-notes.mdx`: Notes Hermes stale base-image state repair. - #6016 -> `docs/reference/commands.mdx`, `docs/reference/commands-nemohermes.mdx`, and `docs/manage-sandboxes/messaging-channels.mdx`: Documents channel policy restoration and rollback on `channels start`. - #5859 -> `docs/about/release-notes.mdx`: Links quickstart network approval guidance. - #5863 -> `docs/about/release-notes.mdx`: Links Teams allowlist guidance in the messaging page. - #5756, #5926, #6010, and #6011 -> `docs/about/release-notes.mdx`: Summarizes the Vitest E2E validation cutover. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: doc-only prose refresh with no runtime behavior change. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [ ] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) `npm run docs` exited 0 and Fern reported one existing light-mode accent contrast warning. `fern check --warnings` confirmed the warning is the site theme contrast ratio, not content introduced by this PR. --- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary Promote the frozen docs-validation behavior into a default-enabled Vitest E2E job before retiring the legacy Nightly workflow. The lane stays deterministic and local-only while preserving selective dispatch, artifacts, and PR result aggregation. ## Related Issue Contributes to NVIDIA#5919. ## Changes - Add a dedicated `docs-validation-vitest` workflow job using the existing live Vitest test. - Keep remote link checks disabled so scheduled results do not depend on external site availability. - Include the job in free-standing selector discovery and PR result aggregation. - Add a focused workflow mutation ratchet for execution, determinism, action pinning, artifacts, and aggregation. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: internal E2E workflow coverage only; user-facing documentation and behavior are unchanged. - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Targeted verification: - `VITEST_MAX_WORKERS=16 npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.ts` - `npm run typecheck:cli` - normal commit and push hooks --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added an end-to-end docs validation workflow boundary check to verify required job behavior, determinism expectations, and step configuration. * Introduced workflow validation coverage for the live docs validation Vitest job, including artifact upload settings. * **New Features** * Added a dedicated live docs validation Vitest scenario that runs and publishes artifacts for inspection. * **Bug Fixes** * Updated PR scenario result reporting so the docs validation job’s pass/fail status is reflected in the PR results table. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
## Summary Refreshes the v0.0.70 release docs from the release announcement and the `v0.0.69..v0.0.70` commit range. It also documents the `channels start` policy restoration behavior that was missing from the shared OpenClaw and Hermes command references, and bumps the Fern CLI version used for docs validation. ## Changes - Replaced the stale `v0.0.70` release-notes entry with the actual release themes, including CLI, onboarding, inference, messaging, Windows, documentation, and release-validation changes. - Documented that `channels start` reapplies the matching built-in network policy preset before rebuild and rolls back to disabled if policy restoration fails. - Bumped `fern/fern.config.json` from `5.55.0` to `5.59.0` for the docs refresh. - Source summary: - NVIDIA#5754 -> `docs/about/release-notes.mdx`: Notes Docker Desktop gateway bridge retry behavior during onboarding. - NVIDIA#5930 -> `docs/about/release-notes.mdx`: Links `nemoclaw use` default sandbox selection to the command reference. - NVIDIA#5948 -> `docs/about/release-notes.mdx`: Links reasoning-compatible endpoint validation to inference documentation. - NVIDIA#5950 -> `docs/about/release-notes.mdx`: Links Windows bootstrap WSL recovery behavior to Windows preparation and troubleshooting docs. - NVIDIA#5856 -> `docs/about/release-notes.mdx`: Notes rebuilt policy preset registry repair. - NVIDIA#5882 and NVIDIA#5949 -> `docs/about/release-notes.mdx`: Notes Hermes stale base-image state repair. - NVIDIA#6016 -> `docs/reference/commands.mdx`, `docs/reference/commands-nemohermes.mdx`, and `docs/manage-sandboxes/messaging-channels.mdx`: Documents channel policy restoration and rollback on `channels start`. - NVIDIA#5859 -> `docs/about/release-notes.mdx`: Links quickstart network approval guidance. - NVIDIA#5863 -> `docs/about/release-notes.mdx`: Links Teams allowlist guidance in the messaging page. - NVIDIA#5756, NVIDIA#5926, NVIDIA#6010, and NVIDIA#6011 -> `docs/about/release-notes.mdx`: Summarizes the Vitest E2E validation cutover. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: doc-only prose refresh with no runtime behavior change. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [ ] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) `npm run docs` exited 0 and Fern reported one existing light-mode accent contrast warning. `fern check --warnings` confirmed the warning is the site theme contrast ratio, not content introduced by this PR. --- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
wscurran
added
area: ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promote the frozen docs-validation behavior into a default-enabled Vitest E2E job before retiring the legacy Nightly workflow. The lane stays deterministic and local-only while preserving selective dispatch, artifacts, and PR result aggregation.
Related Issue
Contributes to #5919.
Changes
docs-validation-vitestworkflow job using the existing live Vitest test.Type of Change
Quality Gates
Verification
Verifiedin GitHubnpx prek run --from-ref main --to-ref HEADpassesnpm testpasses (broad runtime changes only)npm run docsbuilds without warnings (doc changes only)Targeted verification:
VITEST_MAX_WORKERS=16 npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/docs-validation-workflow-boundary.test.tsnpm run typecheck:cliSigned-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit