Skip to content

test(e2e): make live Vitest execution state-safe - #6693

Merged
cv merged 2 commits into
mainfrom
codex/vitest-live-regression-safety
Jul 11, 2026
Merged

test(e2e): make live Vitest execution state-safe#6693
cv merged 2 commits into
mainfrom
codex/vitest-live-regression-safety

Conversation

@cv

@cv cv commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make live Vitest execution state-safe and restore four regression jobs that invoked Vitest without the repository's shared preparation. Live files now run serially against a clean CLI build, while the gateway-drift bypass remains limited to controlled non-live projects.

Related Issue

Part of #6692

Changes

  • Disable file-level parallelism for the stateful e2e-live project while retaining zero whole-test retries.
  • Scope the gateway-drift preflight bypass to the six controlled non-live projects so live and branch-validation runs exercise the real preflight.
  • Clean and rebuild the CLI before the aggregate live E2E command, with a behavior test that proves clean, build, and live-project ordering through stubbed command boundaries.
  • Prepare four regression jobs with the immutable shared E2E action, least-privilege checkout settings, and the correct Vitest project for gateway drift.
  • Document the aggregate live command's build, serialization, and retry behavior.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: focused review confirmed the bypass remains enabled for all six controlled non-live projects, is absent from both live projects, and does not change production preflight behavior; contract coverage is included in this PR and tracked by test: harden Vitest execution, isolation, and feedback loops #6692.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project cli --project integration --project e2e-support src/lib/adapters/openshell/gateway-drift.test.ts test/e2e/support/e2e-live-project-config.test.ts test/regression-e2e-workflow.test.ts --silent=false --reporter=default (38 passed); the final npm-script harness rerun passed 8/8 and npm run source-shape:check reported zero cases.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — env -u SSH_CONNECTION -u SSH_CLIENT npm test -- --reporter=dot (1,418 files and 16,255 tests passed; 2 files and 39 tests skipped). The SSH variables were removed because the host session otherwise activates the dashboard command's intentional SSH hint; its isolated non-SSH rerun passed 15/15.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — passed with 0 errors; Fern reported 2 existing unprinted warnings
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • New Features

    • Added a single command for live end-to-end testing that cleans and rebuilds the CLI automatically.
    • Standardized regression test environment preparation across supported test workflows.
    • Live end-to-end tests now run serially for improved reliability.
  • Bug Fixes

    • Corrected the drift preflight regression suite to run against the integration test project.
  • Documentation

    • Updated live end-to-end testing instructions to use the new aggregate command.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jul 11, 2026
@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 65116bc1-0368-409c-b51e-a9fc0f8f803a

📥 Commits

Reviewing files that changed from the base of the PR and between 4064501 and 47c2616.

📒 Files selected for processing (7)
  • .github/workflows/regression-e2e.yaml
  • package.json
  • test/e2e/docs/MIGRATION.md
  • test/e2e/docs/README.md
  • test/e2e/support/e2e-live-project-config.test.ts
  • test/regression-e2e-workflow.test.ts
  • vitest.config.ts

📝 Walkthrough

Walkthrough

The PR standardizes regression E2E workspace preparation, scopes Vitest environment overrides to non-live projects, serializes live E2E execution, rebuilds the CLI automatically, and adds workflow and command-order regression coverage.

Changes

E2E execution and regression workflow standardization

Layer / File(s) Summary
Vitest project environment and parallelism
vitest.config.ts, test/e2e/support/e2e-live-project-config.test.ts
Non-live projects receive the drift-preflight bypass environment, while live E2E files disable file parallelism and test assertions cover the project configuration.
Live E2E command and validation
package.json, test/e2e/docs/*, test/e2e/support/e2e-live-project-config.test.ts
The aggregate live E2E command cleans and rebuilds the CLI before Vitest; documentation and spawned-command tests verify invocation order and serial execution behavior.
Regression workflow preparation and project selection
.github/workflows/regression-e2e.yaml, test/regression-e2e-workflow.test.ts
Selected regression jobs use read-only contents permissions and shared workspace preparation, while gateway drift preflight runs the integration Vitest project.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

  • NVIDIA/NemoClaw issue 6692 — Covers serialized live E2E execution, fresh CLI builds, controlled drift-bypass environments, and regression workflow setup.

Possibly related PRs

  • NVIDIA/NemoClaw#3423 — Also modifies vitest.config.ts around NEMOCLAW_DISABLE_GATEWAY_DRIFT_PREFLIGHT.

Suggested labels: NV QA, bug-fix, area: e2e

Suggested reviewers: jyaunches, ericksoa

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: making live Vitest E2E execution state-safe.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/vitest-live-regression-safety

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the codex/vitest-live-re... branch remains at 79%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.
File main 4064501 codex/vitest-live-re... 47c2616 +/-
src/lib/security/redact.ts 100% 99% -1%
src/lib/state/m...-acquisition.ts 77% 78% +1%
src/lib/agent/onboard.ts 78% 80% +2%
src/lib/adapter...shell/client.ts 83% 85% +2%
src/lib/onboard...ider-cleanup.ts 82% 85% +3%
src/lib/messagi...ate-resolver.ts 96% 100% +4%
src/lib/messagi...ate-resolver.ts 95% 100% +5%
src/lib/messagi...ate-resolver.ts 88% 94% +6%
src/lib/runner.ts 72% 80% +8%
src/lib/messagi...ate-resolver.ts 78% 89% +11%

Updated July 11, 2026 23:46 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: gateway-drift-preflight-e2e, gateway-health-honest-e2e, openshell-version-pin-e2e, model-router-provider-routed-inference-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No merge-blocking E2E is required. The deterministic risk plan selected no required jobs, and the changed files are limited to E2E CI workflow wiring, Vitest test-project configuration, support tests, package test script behavior, and E2E documentation. They do not modify installer/onboarding implementation, sandbox lifecycle code, credentials handling, security or network policy, inference routing, deployment code, or real assistant user-flow runtime paths.

Optional E2E

  • gateway-drift-preflight-e2e (medium): Optional confidence for the changed regression workflow lane that now prepares the E2E workspace and invokes the gateway drift regression through the integration Vitest project instead of the cli project.
  • gateway-health-honest-e2e (medium): Optional confidence for the changed regression workflow setup path that replaces per-job Node setup with the shared prepare-e2e action before running the live gateway health-honesty regression.
  • openshell-version-pin-e2e (low): Optional confidence for the changed regression workflow setup path using prepare-e2e with build-cli disabled before the OpenShell installer/version-pin regression.
  • model-router-provider-routed-inference-e2e (high): Optional higher-cost confidence for the changed regression workflow setup path before the live Model Router provider-routed inference regression; this requires the NVIDIA API key secret.

New E2E recommendations

  • None.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: No advisor follow-up required beyond maintainer review.
Findings: 0 required · 0 warnings · 0 optional suggestions
Since last review: 0 prior items resolved · 0 still apply · 0 new items found

Workflow run details

This is an automated review. Required findings need action before merge. Warnings and optional suggestions do not require a response or follow-up. A human maintainer makes the final merge decision.

@cv
cv merged commit a31b29b into main Jul 11, 2026
57 checks passed
@cv
cv deleted the codex/vitest-live-regression-safety branch July 11, 2026 23:55
cv added a commit that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Restores `npm run test:fast` as a clean source-backed lane that succeeds
after `dist/` is removed. Compiled CLI contracts now stay in
`package-contract`, and the import guard follows transitive module edges
so compiled artifacts cannot re-enter a fast project through a helper or
alias.

## Related Issue

Part of #6692. Follows #6693, which has merged.

## Changes

- Keep the live gateway-auth scenario's compiled launch and TLS
dependencies at the live test entry while its reusable helper imports
source.
- Discover the command IDs and literal routing metadata needed by source
tests without loading command modules; compiled consumers remain
generated-manifest-only and fail closed when the manifest is absent.
This source-only path is required because command-registry consumers
initialize before command modules can be loaded safely, and
`src/lib/cli/oclif-metadata.test.ts` protects that contract.
- Move Oclif discovery and completion assertions that require built
artifacts into `test/package-contract/`.
- Traverse the exact fast-project roots through static imports, exports,
dynamic imports, `require`, and `createRequire`; report deterministic
shortest chains and reject aliases that resolve into root CLI build
output while ignoring external package layouts.
- Make the doctor flow fixture own its CLI-build prerequisite without
depending on an existing `dist/` tree.
- Keep compiled Oclif inspection branches in a named package-contract
helper and record the live gateway-auth scenario's existing fast parity
owner.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this enforces the existing
contributor contract documented in `CONTRIBUTING.md` and `AGENTS.md`;
the required documentation-writer review found no user-facing change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: the live auth scenario
still receives compiled launch/TLS functions only from its live entry,
and compiled CLI metadata remains generated-manifest-only with a
fail-closed regression test.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npm run test:fast -- --reporter=dot`
(912 files, 9,759 passed, 1 skipped); `npm run test:package --
--reporter=dot` (19 files, 297 passed); boundary guard (34 passed);
doctor flow (10 passed); mock-parity unit (6 passed); live-auth support
(17 passed); import/project checks and CLI typecheck passed; the live
auth test collected from rebuilt artifacts.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — the complete non-live project set passed
in boundary-preserving lanes: `test:fast`, `test:integration` (488
files, 6,206 passed, 38 skipped), and `test:package`.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Improvements**
- Command routing and completion metadata can be discovered from source
without executing command code, with a unified loader and safer handling
when the generated manifest is missing.
- Expanded import-boundary violation reporting with transitive
reachability analysis and an optional violation chain.
- **Bug Fixes**
- Strengthened detection of disallowed module-loading forms and improved
safeguards to avoid unintentionally loading compiled internals.
- Updated sandbox doctor flow mocking to align with runtime path
resolution.
- **Tests**
- Added/updated contract and e2e suites, including a gateway scenario
using injected TLS/launch helpers, new compiled-completion metadata
coverage, and a fail-closed manifest check.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
cv added a commit that referenced this pull request Jul 12, 2026
## Summary
Vitest project validation now compares every filesystem test candidate
with Vitest's complete eight-project roster, rejecting omissions,
overlaps, wrong routing, unexpected files, and unexpected project names.
The shared credential-free E2E workflow now applies Vitest's native
e2e/credential-free tag filter in addition to the existing safe matrix
scanner. The dashboard-port regression fixture also uses a coherent fake
OpenShell 0.0.72 component set and an OS-assigned gateway port so the
repo-wide gate stays hermetic.

This is the final entry in the ordered #6692 stack after #6693, #6696,
#6697, #6699, #6700, #6701, #6702, and #6705 landed.

## Related Issue
Closes #6692

## Changes
- Discover broad test/spec JavaScript and TypeScript candidates under
src, test, and nemoclaw/src, including nested dist, coverage, and .venv
paths that Vitest would see.
- Compare candidate routing with Vitest file listings and the complete
roster from list-tags JSON, including empty and opt-in projects.
- Produce sorted diagnostics for zero membership, overlap, wrong
project, unsupported candidates, unexpected listed files, and missing or
extra projects.
- Invoke Vitest portably through process.execPath and
node_modules/vitest/vitest.mjs.
- Require the exact native credential-free tag filter in the shared E2E
workflow contract while retaining the TypeScript scanner's path,
project, ID, and declaration checks.
- Keep the dashboard-port exhaustion regression independent of port
8080, developer OpenShell overrides, channel selection, and network
installation.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: these are
contributor/test-harness safeguards; AGENTS.md now states the exhaustive
project-membership contract, the E2E README already documents the native
module tag, and the required documentation-writer review found no
user-facing change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent review
covered the shared credential-free workflow and onboarding regression
fixture; it confirmed the filter preserves the existing scanner and the
fixture's child-only environment cannot reach a real OpenShell install.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as Verified in GitHub
- [x] Normal pre-commit, commit-msg, and pre-push hooks passed, or npm
run check:diff passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 46 integration tests passed across the
boundary and dashboard-port files; 4 E2E-support workflow tests passed;
project membership is exact for 1,537 candidates across 8 projects; CLI
type-check, title style, Biome, and diff checks passed. The fixture also
passed with hostile inherited OpenShell paths and dev-channel selection.
- [x] Applicable broad gate passed — npm test passed 1,456 files /
16,515 tests before the final focused hardening; npm run check passed
the repo-wide structural, CLI coverage, and plugin coverage gates,
followed by focused reruns and normal hooks after the final
child-environment pin.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] npm run docs builds without warnings (doc changes only)
- [ ] Doc pages follow the style guide (doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Tests**
- Credential-free end-to-end test runs now execute only tests matching
the credential-free tag.
- Vitest project validation now checks for missing, overlapping,
unsupported, wrong, zero-membership, and unexpected test-to-project
assignments, plus roster mismatches.
- Expanded boundary tests cover test candidate discovery, expected
project mapping, Vitest invocation resolution, and listing/roster
parsing.
- Improved onboarding “dashboard port exhaustion” assertions for more
hermetic, reliable verification.
- **Documentation**
- Updated testing guidance to require Vitest project globs be disjoint
and exhaustive, enforced by the projects check.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Make live Vitest execution state-safe and restore four regression jobs
that invoked Vitest without the repository's shared preparation. Live
files now run serially against a clean CLI build, while the
gateway-drift bypass remains limited to controlled non-live projects.

## Related Issue

Part of NVIDIA#6692

## Changes

- Disable file-level parallelism for the stateful `e2e-live` project
while retaining zero whole-test retries.
- Scope the gateway-drift preflight bypass to the six controlled
non-live projects so live and branch-validation runs exercise the real
preflight.
- Clean and rebuild the CLI before the aggregate live E2E command, with
a behavior test that proves clean, build, and live-project ordering
through stubbed command boundaries.
- Prepare four regression jobs with the immutable shared E2E action,
least-privilege checkout settings, and the correct Vitest project for
gateway drift.
- Document the aggregate live command's build, serialization, and retry
behavior.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: focused review
confirmed the bypass remains enabled for all six controlled non-live
projects, is absent from both live projects, and does not change
production preflight behavior; contract coverage is included in this PR
and tracked by NVIDIA#6692.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project cli
--project integration --project e2e-support
src/lib/adapters/openshell/gateway-drift.test.ts
test/e2e/support/e2e-live-project-config.test.ts
test/regression-e2e-workflow.test.ts --silent=false --reporter=default`
(38 passed); the final npm-script harness rerun passed 8/8 and `npm run
source-shape:check` reported zero cases.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — `env -u SSH_CONNECTION -u SSH_CLIENT npm
test -- --reporter=dot` (1,418 files and 16,255 tests passed; 2 files
and 39 tests skipped). The SSH variables were removed because the host
session otherwise activates the dashboard command's intentional SSH
hint; its isolated non-SSH rerun passed 15/15.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with 0 errors; Fern reported 2 existing unprinted warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a single command for live end-to-end testing that cleans and
rebuilds the CLI automatically.
* Standardized regression test environment preparation across supported
test workflows.
  * Live end-to-end tests now run serially for improved reliability.

* **Bug Fixes**
* Corrected the drift preflight regression suite to run against the
integration test project.

* **Documentation**
* Updated live end-to-end testing instructions to use the new aggregate
command.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Restores `npm run test:fast` as a clean source-backed lane that succeeds
after `dist/` is removed. Compiled CLI contracts now stay in
`package-contract`, and the import guard follows transitive module edges
so compiled artifacts cannot re-enter a fast project through a helper or
alias.

## Related Issue

Part of NVIDIA#6692. Follows NVIDIA#6693, which has merged.

## Changes

- Keep the live gateway-auth scenario's compiled launch and TLS
dependencies at the live test entry while its reusable helper imports
source.
- Discover the command IDs and literal routing metadata needed by source
tests without loading command modules; compiled consumers remain
generated-manifest-only and fail closed when the manifest is absent.
This source-only path is required because command-registry consumers
initialize before command modules can be loaded safely, and
`src/lib/cli/oclif-metadata.test.ts` protects that contract.
- Move Oclif discovery and completion assertions that require built
artifacts into `test/package-contract/`.
- Traverse the exact fast-project roots through static imports, exports,
dynamic imports, `require`, and `createRequire`; report deterministic
shortest chains and reject aliases that resolve into root CLI build
output while ignoring external package layouts.
- Make the doctor flow fixture own its CLI-build prerequisite without
depending on an existing `dist/` tree.
- Keep compiled Oclif inspection branches in a named package-contract
helper and record the live gateway-auth scenario's existing fast parity
owner.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this enforces the existing
contributor contract documented in `CONTRIBUTING.md` and `AGENTS.md`;
the required documentation-writer review found no user-facing change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: the live auth scenario
still receives compiled launch/TLS functions only from its live entry,
and compiled CLI metadata remains generated-manifest-only with a
fail-closed regression test.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npm run test:fast -- --reporter=dot`
(912 files, 9,759 passed, 1 skipped); `npm run test:package --
--reporter=dot` (19 files, 297 passed); boundary guard (34 passed);
doctor flow (10 passed); mock-parity unit (6 passed); live-auth support
(17 passed); import/project checks and CLI typecheck passed; the live
auth test collected from rebuilt artifacts.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — the complete non-live project set passed
in boundary-preserving lanes: `test:fast`, `test:integration` (488
files, 6,206 passed, 38 skipped), and `test:package`.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Improvements**
- Command routing and completion metadata can be discovered from source
without executing command code, with a unified loader and safer handling
when the generated manifest is missing.
- Expanded import-boundary violation reporting with transitive
reachability analysis and an optional violation chain.
- **Bug Fixes**
- Strengthened detection of disallowed module-loading forms and improved
safeguards to avoid unintentionally loading compiled internals.
- Updated sandbox doctor flow mocking to align with runtime path
resolution.
- **Tests**
- Added/updated contract and e2e suites, including a gateway scenario
using injected TLS/launch helpers, new compiled-completion metadata
coverage, and a fail-closed manifest check.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…A#6707)

## Summary
Vitest project validation now compares every filesystem test candidate
with Vitest's complete eight-project roster, rejecting omissions,
overlaps, wrong routing, unexpected files, and unexpected project names.
The shared credential-free E2E workflow now applies Vitest's native
e2e/credential-free tag filter in addition to the existing safe matrix
scanner. The dashboard-port regression fixture also uses a coherent fake
OpenShell 0.0.72 component set and an OS-assigned gateway port so the
repo-wide gate stays hermetic.

This is the final entry in the ordered NVIDIA#6692 stack after NVIDIA#6693, NVIDIA#6696,
NVIDIA#6697, NVIDIA#6699, NVIDIA#6700, NVIDIA#6701, NVIDIA#6702, and NVIDIA#6705 landed.

## Related Issue
Closes NVIDIA#6692

## Changes
- Discover broad test/spec JavaScript and TypeScript candidates under
src, test, and nemoclaw/src, including nested dist, coverage, and .venv
paths that Vitest would see.
- Compare candidate routing with Vitest file listings and the complete
roster from list-tags JSON, including empty and opt-in projects.
- Produce sorted diagnostics for zero membership, overlap, wrong
project, unsupported candidates, unexpected listed files, and missing or
extra projects.
- Invoke Vitest portably through process.execPath and
node_modules/vitest/vitest.mjs.
- Require the exact native credential-free tag filter in the shared E2E
workflow contract while retaining the TypeScript scanner's path,
project, ID, and declaration checks.
- Keep the dashboard-port exhaustion regression independent of port
8080, developer OpenShell overrides, channel selection, and network
installation.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: these are
contributor/test-harness safeguards; AGENTS.md now states the exhaustive
project-membership contract, the E2E README already documents the native
module tag, and the required documentation-writer review found no
user-facing change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent review
covered the shared credential-free workflow and onboarding regression
fixture; it confirmed the filter preserves the existing scanner and the
fixture's child-only environment cannot reach a real OpenShell install.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as Verified in GitHub
- [x] Normal pre-commit, commit-msg, and pre-push hooks passed, or npm
run check:diff passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 46 integration tests passed across the
boundary and dashboard-port files; 4 E2E-support workflow tests passed;
project membership is exact for 1,537 candidates across 8 projects; CLI
type-check, title style, Biome, and diff checks passed. The fixture also
passed with hostile inherited OpenShell paths and dev-channel selection.
- [x] Applicable broad gate passed — npm test passed 1,456 files /
16,515 tests before the final focused hardening; npm run check passed
the repo-wide structural, CLI coverage, and plugin coverage gates,
followed by focused reruns and normal hooks after the final
child-environment pin.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] npm run docs builds without warnings (doc changes only)
- [ ] Doc pages follow the style guide (doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Tests**
- Credential-free end-to-end test runs now execute only tests matching
the credential-free tag.
- Vitest project validation now checks for missing, overlapping,
unsupported, wrong, zero-membership, and unexpected test-to-project
assignments, plus roster mismatches.
- Expanded boundary tests cover test candidate discovery, expected
project mapping, Vitest invocation resolution, and listing/roster
parsing.
- Improved onboarding “dashboard port exhaustion” assertions for more
hermetic, reliable verification.
- **Documentation**
- Updated testing guidance to require Vitest project globs be disjoint
and exhaustive, enforced by the projects check.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@wscurran wscurran added the area: e2e End-to-end tests, nightly failures, or validation infrastructure label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants