Skip to content

perf(test): narrow gateway drift preflight graph - #6286

Merged
cv merged 1 commit into
mainfrom
codex/perf-gateway-preflight-6237
Jul 6, 2026
Merged

perf(test): narrow gateway drift preflight graph#6286
cv merged 1 commit into
mainfrom
codex/perf-gateway-preflight-6237

Conversation

@cv

@cv cv commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Narrows gateway-drift preflight coverage so fail-closed behavior no longer loads the 632-module maintenance and upgrade graph inside a timed hook. This addresses the hottest source-loader path behind #6237 while preserving the distinct process-level drift contracts.

Related Issue

Refs #6237

Changes

  • Centralize sandbox-list preflight, one-shot recovery, result classification, and generic failure handling in a leaf helper shared by backup-all and upgrade-sandboxes.
  • Replace the timed broad-graph preflight suite with leaf behavior tests and small caller-adapter tests, reducing the measured import graph from 632 modules to 68 (89.2%).
  • Retain five distinct compiled-CLI drift sentinels, pin them to case-local OpenShell state, and remove only Vitest's appended TypeScript loader from those compiled children.
  • Verify 36 focused tests in 3.13s locally, including the real process contract in 2.23s; the latest upstream baseline for the removed source test was 9.18s.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: internal test-boundary and shared-helper refactor with no user-facing behavior change; documentation review found no update needed.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent fail-closed/security and performance reviews found no actionable findings; focused coverage verifies preflight ordering, one-shot recovery, retry classification, and generic failure status preservation.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run --project cli --project integration src/lib/openshell-sandbox-list.test.ts src/lib/actions/maintenance.test.ts src/lib/actions/upgrade-sandboxes-preflight.test.ts src/lib/actions/upgrade-sandboxes-recovery.test.ts src/lib/actions/sandbox/rebuild-gateway-drift.test.ts test/gateway-drift-preflight.test.ts --reporter=verbose — 6 files and 36 tests passed in 3.13s.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: final-head CI passed all five CLI coverage shards and the merged cli-tests coverage gate; shard wall times were 5m20s–7m32s with zero infrastructure hook timeouts.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jul 5, 2026
@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR replaces the previous gateway drift preflight/result-issue detection and recovery-based sandbox list validation logic in backupAll and upgradeSandboxes with a single new adapter, captureSandboxListWithGatewayPreflightOrExit, which combines preflight checks, recovery-based capture, and exit handling. Corresponding tests are added and updated, and the old standalone gateway-drift-preflight action test suite is removed.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Changes

Gateway preflight-or-exit consolidation

Layer / File(s) Summary
New preflight-or-exit adapter
src/lib/openshell-sandbox-list.ts, src/lib/openshell-sandbox-list.test.ts
Adds SandboxListPreflightContext type and captureSandboxListWithGatewayPreflightOrExit, which runs preflight checks, captures sandbox list with recovery, and exits on failure; new test suite covers drift exit, success, retry, protobuf mismatch ordering, and non-recoverable failures.
backupAll wiring
src/lib/actions/maintenance.ts, src/lib/actions/maintenance.test.ts
backupAll() now calls the new adapter instead of manual preflight/result/recovery branching; tests updated to mock the adapter across preflight-gating, ready-sandbox filtering, and error scenarios.
upgradeSandboxes wiring
src/lib/actions/upgrade-sandboxes.ts, src/lib/actions/upgrade-sandboxes-preflight.test.ts, src/lib/actions/upgrade-sandboxes-recovery.test.ts
upgradeSandboxes obtains liveResult via the new adapter; new preflight-focused test suite and updated recovery tests mock this call.
E2E harness updates
test/gateway-drift-preflight.test.ts, src/lib/actions/gateway-drift-preflight.test.ts
CLI env sanitization strips a source-loader NODE_OPTIONS flag; host-process test helper split into live and markerless variants; protobuf-mismatch scenario asserts non-zero exit status; the older standalone gateway-drift-preflight test suite is removed.

Suggested labels: refactor

Suggested reviewers: jyaunches, ericksoa, prekshivyas

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: narrowing gateway drift preflight coverage and reducing the test graph.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/perf-gateway-preflight-6237

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/perf-gateway-p... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-gateway-p... 270037a +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the codex/perf-gateway-p... branch is 71%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-gateway-p... 270037a +/-
src/lib/shields...nsition-lock.ts 87%
src/lib/actions...all/run-plan.ts 81%
src/lib/state/o...oard-session.ts 81%
src/lib/onboard/preflight.ts 71%
src/lib/state/sandbox.ts 71%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/shields/index.ts 68%
src/lib/policy/index.ts 66%
src/lib/actions...licy-channel.ts 60%
src/lib/onboard.ts 24%

Updated July 05, 2026 19:44 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 2 suggestions · 3 test follow-ups
Since last review: 0 prior items resolved · 0 still apply · 2 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: E2E coverage for upgrade-sandboxes gateway preflight removed
  • PRA-1 In-scope improvement: E2E coverage for upgrade-sandboxes gateway preflight removed in test/gateway-drift-preflight.test.ts:1
  • PRA-2 In-scope improvement: process.exit in shared adapter makes unit testing slightly awkward in src/lib/openshell-sandbox-list.ts:73

Findings index

ID Severity Category Location Required action
PRA-1 Improvement tests test/gateway-drift-preflight.test.ts:1 Add an E2E test case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add a comment in upgrade-sandboxes-preflight.test.ts explaining why unit test coverage is sufficient (shared adapter tested via backup-all E2E).
PRA-2 Improvement architecture src/lib/openshell-sandbox-list.ts:73 Consider returning a Result<Error, SandboxListResult> type in a future refactor for better testability. No change needed in this PR — current mock pattern is correct and well-tested.
Review findings by urgency: 0 required fixes, 0 items to resolve/justify, 2 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

  • None.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-1 Improvement — E2E coverage for upgrade-sandboxes gateway preflight removed

  • Location: test/gateway-drift-preflight.test.ts:1
  • Category: tests
  • Problem: The E2E test for `upgrade-sandboxes --check` host-process drift was removed during the test restructuring. Only the new unit test (src/lib/actions/upgrade-sandboxes-preflight.test.ts) covers this path. Since the gateway preflight adapter is shared and tested via backup-all E2E, the risk is low, but explicit E2E coverage for upgrade-sandboxes would improve confidence.
  • Impact: If the upgrade-sandboxes caller passes incorrect context to the shared adapter, it would not be caught by E2E tests. Unit test mocks the adapter call, so integration issues between upgrade-sandboxes and the adapter are not validated at runtime.
  • Suggested action: Add an E2E test case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add a comment in upgrade-sandboxes-preflight.test.ts explaining why unit test coverage is sufficient (shared adapter tested via backup-all E2E).
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run `pnpm vitest run test/gateway-drift-preflight.test.ts` — only backup-all cases tested. Run `pnpm vitest run src/lib/actions/upgrade-sandboxes-preflight.test.ts` — unit tests cover upgrade-sandboxes preflight via mocked adapter.
  • Missing regression test: Add E2E case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add comment in upgrade-sandboxes-preflight.test.ts documenting that shared adapter E2E coverage via backup-all is sufficient.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: PR diff shows test/gateway-drift-preflight.test.ts removed host-process-upgrade case; new upgrade-sandboxes-preflight.test.ts has 3 unit tests covering preflight adapter.

PRA-2 Improvement — process.exit in shared adapter makes unit testing slightly awkward

  • Location: src/lib/openshell-sandbox-list.ts:73
  • Category: architecture
  • Problem: captureSandboxListWithGatewayPreflightOrExit calls process.exit directly on failure, requiring tests to mock process.exit to throw an Error. This works but adds boilerplate and couples the adapter to process lifecycle.
  • Impact: Minor test ergonomics issue. All current tests correctly mock process.exit and verify behavior. No production impact.
  • Suggested action: Consider returning a Result<Error, SandboxListResult> type in a future refactor for better testability. No change needed in this PR — current mock pattern is correct and well-tested.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check src/lib/openshell-sandbox-list.test.ts — tests mock process.exit to throw Error('process.exit(1)') and catch rejection to verify fail-closed behavior.
  • Missing regression test: No regression test needed — current mock pattern works correctly. Future refactor to Result type would simplify tests.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Line 73 in openshell-sandbox-list.ts calls process.exit(1) on preflight failure; line 79 calls process.exit(1) on result drift; line 86 calls process.exit on generic failure.
Simplification opportunities: 1 possible cut

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-2 shrink (src/lib/openshell-sandbox-list.ts:73): Direct process.exit calls in captureSandboxListWithGatewayPreflightOrExit
    • Replacement: Return Result<Error, SandboxListResult> and let callers handle exit
    • Net: 5 lines
    • Safety boundary: Fail-closed behavior must remain: callers must still exit on error. The adapter must not swallow drift/failure signals.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add E2E test case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or document rationale for unit-only coverage. Static tests comprehensive (36 tests covering all branches of new adapter + caller adapters). Runtime validation gap: E2E for upgrade-sandboxes gateway preflight removed (only unit test remains). Shared adapter tested via backup-all E2E mitigates risk.
  • PRA-T2 Runtime validation — Consider integration test for empty-registry early return (currently only unit tests). Static tests comprehensive (36 tests covering all branches of new adapter + caller adapters). Runtime validation gap: E2E for upgrade-sandboxes gateway preflight removed (only unit test remains). Shared adapter tested via backup-all E2E mitigates risk.
  • PRA-T3 E2E coverage for upgrade-sandboxes gateway preflight removed — Add an E2E test case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add a comment in upgrade-sandboxes-preflight.test.ts explaining why unit test coverage is sufficient (shared adapter tested via backup-all E2E).
Since last review details

Current findings, using the urgency labels above:

PRA-1 Improvement — E2E coverage for upgrade-sandboxes gateway preflight removed

  • Location: test/gateway-drift-preflight.test.ts:1
  • Category: tests
  • Problem: The E2E test for `upgrade-sandboxes --check` host-process drift was removed during the test restructuring. Only the new unit test (src/lib/actions/upgrade-sandboxes-preflight.test.ts) covers this path. Since the gateway preflight adapter is shared and tested via backup-all E2E, the risk is low, but explicit E2E coverage for upgrade-sandboxes would improve confidence.
  • Impact: If the upgrade-sandboxes caller passes incorrect context to the shared adapter, it would not be caught by E2E tests. Unit test mocks the adapter call, so integration issues between upgrade-sandboxes and the adapter are not validated at runtime.
  • Suggested action: Add an E2E test case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add a comment in upgrade-sandboxes-preflight.test.ts explaining why unit test coverage is sufficient (shared adapter tested via backup-all E2E).
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run `pnpm vitest run test/gateway-drift-preflight.test.ts` — only backup-all cases tested. Run `pnpm vitest run src/lib/actions/upgrade-sandboxes-preflight.test.ts` — unit tests cover upgrade-sandboxes preflight via mocked adapter.
  • Missing regression test: Add E2E case for upgrade-sandboxes host-process drift in test/gateway-drift-preflight.test.ts, or add comment in upgrade-sandboxes-preflight.test.ts documenting that shared adapter E2E coverage via backup-all is sufficient.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: PR diff shows test/gateway-drift-preflight.test.ts removed host-process-upgrade case; new upgrade-sandboxes-preflight.test.ts has 3 unit tests covering preflight adapter.

PRA-2 Improvement — process.exit in shared adapter makes unit testing slightly awkward

  • Location: src/lib/openshell-sandbox-list.ts:73
  • Category: architecture
  • Problem: captureSandboxListWithGatewayPreflightOrExit calls process.exit directly on failure, requiring tests to mock process.exit to throw an Error. This works but adds boilerplate and couples the adapter to process lifecycle.
  • Impact: Minor test ergonomics issue. All current tests correctly mock process.exit and verify behavior. No production impact.
  • Suggested action: Consider returning a Result<Error, SandboxListResult> type in a future refactor for better testability. No change needed in this PR — current mock pattern is correct and well-tested.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check src/lib/openshell-sandbox-list.test.ts — tests mock process.exit to throw Error('process.exit(1)') and catch rejection to verify fail-closed behavior.
  • Missing regression test: No regression test needed — current mock pattern works correctly. Future refactor to Result type would simplify tests.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Line 73 in openshell-sandbox-list.ts calls process.exit(1) on preflight failure; line 79 calls process.exit(1) on result drift; line 86 calls process.exit on generic failure.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 6 test follow-ups
Top item: No actionable findings

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Acceptance clause
  • PRA-T6 Add or justify test follow-up: Acceptance clause
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Process-level `nemoclaw backup-all` exits before `openshell sandbox list` when host-process gateway drift is detected. The changed production files sit on runtime/sandbox maintenance paths. Unit coverage is strong for the shared helper and caller delegation, and the changed process-level E2E still covers backup-all drift sentinels, but read-only review did not execute the tests.
  • PRA-T2 Runtime validation — Process-level `nemoclaw backup-all` surfaces protobuf/schema mismatch without treating the sandbox as stopped. The changed production files sit on runtime/sandbox maintenance paths. Unit coverage is strong for the shared helper and caller delegation, and the changed process-level E2E still covers backup-all drift sentinels, but read-only review did not execute the tests.
  • PRA-T3 Runtime validation — Process-level `nemoclaw upgrade-sandboxes --check` exits before classification/rebuild when host-process gateway drift is detected. The changed production files sit on runtime/sandbox maintenance paths. Unit coverage is strong for the shared helper and caller delegation, and the changed process-level E2E still covers backup-all drift sentinels, but read-only review did not execute the tests.
  • PRA-T4 Runtime validation — `captureSandboxListWithGatewayPreflightOrExit` preserves retry failure exit status after one successful gateway recovery attempt. The changed production files sit on runtime/sandbox maintenance paths. Unit coverage is strong for the shared helper and caller delegation, and the changed process-level E2E still covers backup-all drift sentinels, but read-only review did not execute the tests.
  • PRA-T5 Acceptance clause — Replace the timed broad-graph preflight suite with leaf behavior tests and small caller-adapter tests, reducing the measured import graph from 632 modules to 68 (89.2%). — add test evidence or identify existing coverage. `src/lib/actions/gateway-drift-preflight.test.ts` is deleted and replaced/covered by `src/lib/openshell-sandbox-list.test.ts`, `src/lib/actions/maintenance.test.ts`, and `src/lib/actions/upgrade-sandboxes-preflight.test.ts`. The import-graph reduction number is a PR-body claim and was not independently verified in this read-only review.
  • PRA-T6 Acceptance clause — Verify 36 focused tests in 3.13s locally, including the real process contract in 2.23s; the latest upstream baseline for the removed source test was 9.18s. — add test evidence or identify existing coverage. The changed test files contain focused coverage for the claimed behavior, but this read-only review did not execute tests or verify timing/performance claims.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: gateway-drift-preflight, state-backup-restore, upgrade-stale-sandbox
Optional E2E: openshell-gateway-upgrade, cloud-onboard

Dispatch hint: gateway-drift-preflight,state-backup-restore,upgrade-stale-sandbox

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • gateway-drift-preflight (low): Directly validates the changed gateway drift preflight/fail-closed path using the real CLI with PATH-resolved openshell/docker shims, including image drift, protobuf mismatch, host-process drift, and ensuring unsafe sandbox state mutation is not attempted.
  • state-backup-restore (high): backup-all behavior changed in a runtime path that gates pre-upgrade state safety. This live job exercises real backup/restore lifecycle behavior against OpenShell and should catch regressions in listing Ready sandboxes and preserving state.
  • upgrade-stale-sandbox (high): upgrade-sandboxes live behavior changed before stale classification and rebuild. This job proves upgrade-sandboxes --check detects stale sandboxes, rebuilds them, and reports clean state afterward through the real OpenShell/sandbox lifecycle.

Optional E2E

  • openshell-gateway-upgrade (high): Useful adjacent confidence because the changed backup-all and upgrade-sandboxes paths participate in installer/gateway upgrade and survivor sandbox recovery flows, but the direct risk is covered by gateway-drift-preflight plus the backup and stale-upgrade jobs.
  • cloud-onboard (high): Optional full hosted onboarding confidence if maintainers want a broad end-to-end smoke after changing shared OpenShell sandbox-list/gateway handling used by lifecycle commands; this PR does not touch onboard state-machine resume paths, so it is not required by the onboarding resume rule.

New E2E recommendations

  • gateway drift preflight for upgrade-sandboxes (medium): The updated test/gateway-drift-preflight.test.ts no longer appears to exercise the upgrade-sandboxes host-process drift preflight path, even though upgrade-sandboxes now relies on the same shared preflight helper. Add a scenario that runs nemoclaw upgrade-sandboxes --check with host-process gateway drift and asserts no sandbox list or classification occurs.
    • Suggested test: Extend the existing gateway-drift-preflight E2E job/test with an upgrade-sandboxes --check host-process drift case.

Dispatch hint

  • Workflow: e2e.yaml
  • jobs input: gateway-drift-preflight,state-backup-restore,upgrade-stale-sandbox

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: upgrade-stale-sandbox, openshell-gateway-upgrade
Optional E2E targets: None

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=upgrade-stale-sandbox
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=openshell-gateway-upgrade

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • upgrade-stale-sandbox: The PR changes the live upgrade-sandboxes path and its shared OpenShell sandbox-list gateway preflight/recovery adapter; this job runs the discrete live upgrade-sandboxes regression against real Docker/OpenShell.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=upgrade-stale-sandbox
  • openshell-gateway-upgrade: The PR changes backup-all and the shared sandbox-list preflight/recovery path used by the installer’s OpenShell gateway upgrade flow; this live job exercises the pre-upgrade backup and upgrade handoff with real Docker/OpenShell state.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=openshell-gateway-upgrade

Optional E2E targets

  • None.

Relevant changed files

  • src/lib/actions/maintenance.ts
  • src/lib/actions/upgrade-sandboxes.ts
  • src/lib/openshell-sandbox-list.ts

@cv

cv commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator Author

Advisor follow-up for the final head (270037a):

  • PRA-T1, PRA-T2, PRA-T4, PRA-T5, and PRA-T6 are covered by the focused command recorded in the PR body: 36/36 tests passed, including the retained process-level host-drift and protobuf sentinels, the retry-status unit case, and the replacement leaf/caller suites. The graph count was measured deterministically from the two import roots (632 modules versus 68); the timing claims are local measurements and are intentionally not presented as satisfying perf(ci): eliminate source-loader contention and hook timeouts in CLI coverage shards #6237's constrained-x64 acceptance criteria.
  • For PRA-1 / PRA-T3, the removed upgrade-sandboxes --check child duplicated the same shared fail-closed adapter already exercised by the retained backup-all process sentinel. src/lib/actions/upgrade-sandboxes-preflight.test.ts separately asserts the exact upgrade action/command context and proves classification/rebuild are not reached when the adapter exits. Re-adding the redundant child would increase the process-heavy suite this PR is narrowing without covering a distinct shared-helper branch, so the layered process + caller coverage is intentional.
  • For PRA-T2 (empty registry), both callers have direct early-return tests before the shared helper. A third process-level assertion would duplicate a pure caller branch and add another CLI startup to the performance-sensitive file.
  • PRA-2 is deferred: moving process exit responsibility back into both callers would broaden this performance-focused refactor and duplicate a security-sensitive fail-closed policy that is now centralized and exhaustively tested.

No advisor item requires a code change on this head.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
test/gateway-drift-preflight.test.ts (1)

231-237: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Suffix-only stripping is fragile.

nodeOptionsWithoutSourceLoader only removes SOURCE_REQUIRE_OPTION when it appears at the exact tail of NODE_OPTIONS (or equals it entirely). If any other flag is appended after the source loader (e.g., by another test-runner integration or future ambient env changes), the loader silently leaks into the compiled-dist child process, undermining the intent of "pins them to case-local OpenShell state" and removing the source loader from compiled children.

♻️ More robust token-based stripping
 function nodeOptionsWithoutSourceLoader(nodeOptions: string | undefined): string {
-  if (!nodeOptions || nodeOptions === SOURCE_REQUIRE_OPTION) return "";
-  const sourceLoaderSuffix = ` ${SOURCE_REQUIRE_OPTION}`;
-  return nodeOptions.endsWith(sourceLoaderSuffix)
-    ? nodeOptions.slice(0, -sourceLoaderSuffix.length)
-    : nodeOptions;
+  if (!nodeOptions) return "";
+  return nodeOptions
+    .split(/\s+/)
+    .filter((token) => token && token !== SOURCE_REQUIRE_OPTION)
+    .join(" ");
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/gateway-drift-preflight.test.ts` around lines 231 - 237,
`nodeOptionsWithoutSourceLoader` is too brittle because it only strips
`SOURCE_REQUIRE_OPTION` when it is the exact suffix of `NODE_OPTIONS`; update
this helper to remove that token by parsing `nodeOptions` into individual flags
and filtering out the source-loader entry wherever it appears, then rejoin the
remaining options. Keep the existing `SOURCE_REQUIRE_OPTION` symbol and adjust
the logic used by the child-process setup in
`test/gateway-drift-preflight.test.ts` so compiled-dist children never inherit
the loader even when other `NODE_OPTIONS` flags are present.
src/lib/openshell-sandbox-list.test.ts (1)

126-164: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use an argument-based detectResultIssue mock here.
mockReturnValueOnce(null).mockReturnValueOnce(issue) ties this test to the current two-call sequence and can leave queued state behind if that flow changes. A value-based mockImplementation keeps the assertion focused on the retry result instead of call order.

♻️ Suggested fix
-    mocks.detectResultIssue.mockReturnValueOnce(null).mockReturnValueOnce(issue);
+    mocks.detectResultIssue.mockImplementation((result: { output?: string }) =>
+      result.output === issue.output ? issue : null,
+    );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/openshell-sandbox-list.test.ts` around lines 126 - 164, The test for
captureSandboxListWithGatewayPreflightOrExit is relying on a queued
detectResultIssue return sequence, which couples it to the current call order.
Update the mock in the protobuf_mismatch case to use an argument-based
mockImplementation that returns the issue only for the retry output and null
otherwise, so the assertion stays focused on the observed result rather than the
number of detectResultIssue calls. Keep the rest of the test behavior and
expectations unchanged.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/openshell-sandbox-list.test.ts`:
- Around line 126-164: The test for captureSandboxListWithGatewayPreflightOrExit
is relying on a queued detectResultIssue return sequence, which couples it to
the current call order. Update the mock in the protobuf_mismatch case to use an
argument-based mockImplementation that returns the issue only for the retry
output and null otherwise, so the assertion stays focused on the observed result
rather than the number of detectResultIssue calls. Keep the rest of the test
behavior and expectations unchanged.

In `@test/gateway-drift-preflight.test.ts`:
- Around line 231-237: `nodeOptionsWithoutSourceLoader` is too brittle because
it only strips `SOURCE_REQUIRE_OPTION` when it is the exact suffix of
`NODE_OPTIONS`; update this helper to remove that token by parsing `nodeOptions`
into individual flags and filtering out the source-loader entry wherever it
appears, then rejoin the remaining options. Keep the existing
`SOURCE_REQUIRE_OPTION` symbol and adjust the logic used by the child-process
setup in `test/gateway-drift-preflight.test.ts` so compiled-dist children never
inherit the loader even when other `NODE_OPTIONS` flags are present.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3572c6d9-85c0-458f-bb54-be6f10096125

📥 Commits

Reviewing files that changed from the base of the PR and between e67c901 and 270037a.

📒 Files selected for processing (9)
  • src/lib/actions/gateway-drift-preflight.test.ts
  • src/lib/actions/maintenance.test.ts
  • src/lib/actions/maintenance.ts
  • src/lib/actions/upgrade-sandboxes-preflight.test.ts
  • src/lib/actions/upgrade-sandboxes-recovery.test.ts
  • src/lib/actions/upgrade-sandboxes.ts
  • src/lib/openshell-sandbox-list.test.ts
  • src/lib/openshell-sandbox-list.ts
  • test/gateway-drift-preflight.test.ts
💤 Files with no reviewable changes (1)
  • src/lib/actions/gateway-drift-preflight.test.ts

@cv
cv merged commit 8770d06 into main Jul 6, 2026
52 checks passed
@cv
cv deleted the codex/perf-gateway-preflight-6237 branch July 6, 2026 01:16
cv added a commit that referenced this pull request Jul 7, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative #6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes #6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative #6245 acceptance path after #6276/#6336/#6383
converted the named onboarding hotspots, #6285/#6417 retained
representative process contracts, and #6286/#6299/#6388/#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. #6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Narrows gateway-drift preflight coverage so fail-closed behavior no
longer loads the 632-module maintenance and upgrade graph inside a timed
hook. This addresses the hottest source-loader path behind NVIDIA#6237 while
preserving the distinct process-level drift contracts.

## Related Issue
Refs NVIDIA#6237

## Changes
- Centralize sandbox-list preflight, one-shot recovery, result
classification, and generic failure handling in a leaf helper shared by
`backup-all` and `upgrade-sandboxes`.
- Replace the timed broad-graph preflight suite with leaf behavior tests
and small caller-adapter tests, reducing the measured import graph from
632 modules to 68 (89.2%).
- Retain five distinct compiled-CLI drift sentinels, pin them to
case-local OpenShell state, and remove only Vitest's appended TypeScript
loader from those compiled children.
- Verify 36 focused tests in 3.13s locally, including the real process
contract in 2.23s; the latest upstream baseline for the removed source
test was 9.18s.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: internal test-boundary and
shared-helper refactor with no user-facing behavior change;
documentation review found no update needed.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent
fail-closed/security and performance reviews found no actionable
findings; focused coverage verifies preflight ordering, one-shot
recovery, retry classification, and generic failure status preservation.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run --project cli --project integration
src/lib/openshell-sandbox-list.test.ts
src/lib/actions/maintenance.test.ts
src/lib/actions/upgrade-sandboxes-preflight.test.ts
src/lib/actions/upgrade-sandboxes-recovery.test.ts
src/lib/actions/sandbox/rebuild-gateway-drift.test.ts
test/gateway-drift-preflight.test.ts --reporter=verbose` — 6 files and
36 tests passed in 3.13s.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: final-head CI passed all
five CLI coverage shards and the merged `cli-tests` coverage gate; shard
wall times were 5m20s–7m32s with zero infrastructure hook timeouts.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative NVIDIA#6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes NVIDIA#6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative NVIDIA#6245 acceptance path after NVIDIA#6276/NVIDIA#6336/NVIDIA#6383
converted the named onboarding hotspots, NVIDIA#6285/NVIDIA#6417 retained
representative process contracts, and NVIDIA#6286/NVIDIA#6299/NVIDIA#6388/NVIDIA#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. NVIDIA#6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@wscurran wscurran added the area: ci CI workflows, checks, release automation, or GitHub Actions label Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants