Skip to content

fix: reuse gateway NVIDIA credential for non-interactive recreate - #5454

Merged
cv merged 4 commits into
mainfrom
auto/fix-5441-dgx-spark-onboard-nemoclaw-onboard-recreate
Jun 17, 2026
Merged

fix: reuse gateway NVIDIA credential for non-interactive recreate#5454
cv merged 4 commits into
mainfrom
auto/fix-5441-dgx-spark-onboard-nemoclaw-onboard-recreate

Conversation

@jason-ma-nv

@jason-ma-nv jason-ma-nv commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #5441: non-interactive nemoclaw onboard --recreate-sandbox aborting at [3/8] with "NVIDIA Endpoints endpoint validation failed". When the NVIDIA Endpoints ("build") provider is recovered from an existing sandbox's saved gateway state, the OpenShell gateway already holds the validated credential and nothing is written to disk, so the host process has no local API key. The credential check in setupNim already allowed the flow to continue (provider exists in gateway), but the subsequent endpoint validation probe called getCredential('NVIDIA_INFERENCE_API_KEY'), got null, and probed the endpoint unauthenticated — which fails and exits at [3/8]. The fix reuses the gateway's already-validated credential and skips the re-validation probe in that branch.

Related Issue

Fixes #5441

Changes

  • src/lib/onboard.ts: when the non-interactive build flow has no local key but the gateway holds the credential, reuse it and skip endpoint re-validation; otherwise probe as before. Defaults preferredInferenceApi to openai-completions (consistent with the build provider's normal probe result).
  • src/lib/onboard/build-credential-reuse.ts (new): extracted resolveNonInteractiveBuildCredential (credential gate → reuse flag) and resolveBuildPreferredInferenceApi (skip-or-probe decision) so src/lib/onboard.ts stays net-neutral/smaller per the codebase growth guardrail. The growth guardrail allows growth under src/lib/onboard/**; onboard.ts is net −5 lines vs main.
  • Removed the now-unused logMissingNvidiaApiKeyHelp import from onboard.ts (the helper imports its own copy).
  • test/onboard-build-recreate-credential-reuse.test.ts (new): regression guard that drives the exported setupNim() in a child process with a fake openshell reporting an nvidia-prod gateway route and no local key; asserts setupNim resolves (exit 0), logs the skip message, selects nvidia-prod, and never probes /chat/completions.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Verification details reported by Claude Code:

  • npm run build:cli — clean.
  • npm run typecheck:cli — no new errors in src/lib/onboard.ts or the new helper.
  • npx vitest run test/onboard-build-recreate-credential-reuse.test.ts test/onboard-inference-smoke.test.ts test/onboard-resume-provider-recovery.test.ts — 25 tests pass.
  • npx biome check --write on the changed files — no fixes needed.
  • src/lib/onboard.ts is net +32/−37 (−5) vs main, satisfying the codebase-growth-guardrails check that previously failed.

Signed-off-by: Jason Ma jama@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved non-interactive “build” onboarding credential reuse to enforce local-key requirements in the right scenarios, and refined preferred inference API selection with retry/reuse handling.
    • Added a controlled way to skip host inference smoke verification to reduce unnecessary endpoint probing during sandbox credential recreation.
  • New Features

    • Introduced skipHostInferenceSmoke in onboarding provider/inference flow to optionally bypass host inference smoke checks.
  • Tests

    • Added end-to-end regression tests for “build recreate” credential reuse (reuse success vs required-local-key failure).
    • Updated NIM/onboarding selection test fixtures and assertions to include the new flag.

Non-interactive `onboard --recreate-sandbox` recovers the NVIDIA Endpoints
('build') provider from the existing sandbox's gateway state. The gateway
already holds the validated credential and nothing is written to disk, so the
host process has no local API key. The flow correctly continued past the
credential check (provider exists in gateway) but then ran the endpoint
validation probe unauthenticated, failing at [3/8] with 'endpoint validation
failed'. Skip the re-validation probe when reusing a gateway-only credential,
so recreate proceeds through all stages.

Fixes #5441

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jason-ma-nv jason-ma-nv self-assigned this Jun 15, 2026
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 657db226-d9e9-49c8-a61f-2924bab006b2

📥 Commits

Reviewing files that changed from the base of the PR and between 45133f5 and e41eff8.

📒 Files selected for processing (2)
  • src/lib/onboard.ts
  • src/lib/onboard/setup-nim-selection.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/lib/onboard/setup-nim-selection.ts

📝 Walkthrough

Walkthrough

This PR adds a skipHostInferenceSmoke flag throughout the inference onboarding flow to conditionally bypass endpoint validation smoke tests. It implements non-interactive credential reuse logic for "build" provider selection via helper functions, threads the flag through state initialization, handler contracts, and orchestration, and wires conditional smoke-test skipping into setupInference. Regression tests validate that non-interactive "build" mode with existing gateway credentials skips endpoint probing and succeeds, while explicit "build" selection without local keys fails pre-validation.

Changes

Conditional smoke testing feature and build credential reuse

Layer / File(s) Summary
State model: skipHostInferenceSmoke field
src/lib/onboard/setup-nim-selection.ts, src/lib/onboard/setup-nim-ollama.ts, src/lib/onboard/setup-nim-selection.test.ts, src/lib/onboard/setup-nim-ollama.test.ts
SetupNimSelectionState gains skipHostInferenceSmoke: boolean field; it is initialized to false in cloud fallback, Ollama configuration, NIM selection paths, and test fixtures to establish consistent state shape.
Handler contract and provider-inference integration
src/lib/onboard/machine/handlers/provider-inference.ts
ProviderSelectionResult includes optional skipHostInferenceSmoke field; setupInference contract extends to accept the option; handler tracks and captures the flag from selection result, then builds conditional inferenceOptions to pass both allowToolsIncompatible and skipHostInferenceSmoke to setupInference in resume and non-resume paths.
Build credential reuse: validation and API resolution
src/lib/onboard/build-credential-reuse.ts
New module exports resolveNonInteractiveBuildCredential, which validates local NVIDIA key presence and gates gateway credential reuse based on recoveredFromSandbox flag, and resolveBuildPreferredInferenceApi, which either bypasses endpoint re-validation or loops on probe-based validation based on reuse decision.
Onboard.ts orchestration: credential reuse wiring
src/lib/onboard.ts
Imports credential reuse helpers; initializes skipHostInferenceSmoke to false in setupNim; applies credential reuse logic in non-interactive "build" mode; threads the flag through setupNim return type and state exchanges with handler.
setupInference: conditional smoke test bypass
src/lib/onboard.ts
setupInference conditionally skips verifyOnboardInferenceSmoke when options.skipHostInferenceSmoke === true; otherwise retains existing smoke validation behavior.
Regression tests: credential reuse and smoke skip
test/onboard-build-recreate-credential-reuse.test.ts
Vitest subprocess tests for issue #5441: non-interactive recreate-sandbox with no local keys succeeds, reuses gateway credential, and skips endpoint smoke probing; explicit "build" selection without local keys fails pre-validation before any probing.

Sequence Diagram

sequenceDiagram
    participant User as User/CI
    participant OnboardFlow as onboard.ts<br/>(setupNim)
    participant ResolveCredential as resolveNonInteractive<br/>BuildCredential
    participant ResolveAPI as resolveBuild<br/>PreferredInferenceApi
    participant ValidateEP as validateOpenAiLike<br/>Selection (probe)
    participant SetupInf as setupInference
    participant VerifySmoke as verifyOnboard<br/>InferenceSmoke
    
    User->>OnboardFlow: Call setupNim (non-interactive "build")
    OnboardFlow->>ResolveCredential: Check local key & gateway existence<br/>(recoveredFromSandbox=true)
    ResolveCredential->>OnboardFlow: Return reuseGatewayCredentialWithoutLocalKey=true
    OnboardFlow->>ResolveAPI: Resolve preferred API with reuse flag
    alt reuseGatewayCredentialWithoutLocalKey is true
        ResolveAPI->>OnboardFlow: Return "openai-completions"<br/>(skip probe loop)
    else reuseGatewayCredentialWithoutLocalKey is false
        ResolveAPI->>ValidateEP: Loop probe() until validation succeeds
        ValidateEP->>ResolveAPI: Return validated API
    end
    OnboardFlow->>OnboardFlow: Set skipHostInferenceSmoke=true<br/>(when credential reused)
    OnboardFlow->>SetupInf: Call setupInference<br/>with skipHostInferenceSmoke option
    alt skipHostInferenceSmoke === true
        SetupInf->>SetupInf: Skip smoke verification
    else skipHostInferenceSmoke is false
        SetupInf->>VerifySmoke: Validate endpoint
        VerifySmoke->>SetupInf: Endpoint probe result
    end
    SetupInf->>User: Return (success or error)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Possibly related PRs

  • NVIDIA/NemoClaw#5173 — The main PR's build-provider non-interactive credential reuse and skipHostInferenceSmoke logic depends on recoveredFromSandbox produced by the refactored provider-selection flow from this PR.
  • NVIDIA/NemoClaw#5420 — Both PRs modify the onboarding selection-state flow in src/lib/onboard/setup-nim-selection.ts; the retrieved PR refactors selection into that module and the main PR extends the shared state with skipHostInferenceSmoke.
  • NVIDIA/NemoClaw#5432 — Both PRs modify the "Setup Nim" Ollama onboarding flow (src/lib/onboard/setup-nim-ollama.ts and its tests), with the main PR adding/initializing skipHostInferenceSmoke alongside the earlier Ollama handler/state-shape refactor.

Suggested labels

bug-fix, area: onboarding

Suggested reviewers

  • cv
  • prekshivyas

Poem

🐇 A gateway credential, once trusted and true,
Need not be tested again through and through!
The bunny refactored the smoke's weary test,
Skip validation, let reuse do its best.
Build paths now flourish, no curl loops in sight,
Credentials reused—onboarding feels right! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and concisely describes the main fix: reusing the gateway NVIDIA credential for non-interactive recreate-sandbox flow, which is the core issue addressed in PR #5441.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch auto/fix-5441-dgx-spark-onboard-nemoclaw-onboard-recreate

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-code-quality

github-code-quality Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the auto/fix-5441-dgx-sp... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main auto/fix-5441-dgx-sp... e41eff8 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the auto/fix-5441-dgx-sp... branch is 46%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main auto/fix-5441-dgx-sp... e41eff8 +/-
src/lib/state/o...oard-session.ts 90%
src/lib/inference/local.ts 76%
src/lib/sandbox/config.ts 72%
src/lib/actions...dbox/rebuild.ts 67%
src/lib/onboard/preflight.ts 64%
src/lib/actions...licy-channel.ts 56%
src/lib/state/sandbox.ts 55%
src/lib/onboard...er-gpu-patch.ts 50%
src/lib/policy/index.ts 49%
src/lib/onboard.ts 18%

Updated June 17, 2026 18:20 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: onboard-resume-vitest, double-onboard-vitest, cloud-onboard-vitest
Optional E2E: inference-routing-vitest, credential-migration-vitest

Dispatch hint: onboard-resume-vitest,double-onboard-vitest,cloud-onboard-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • onboard-resume-vitest (medium): Exercises the real non-interactive onboard resume path with NVIDIA_INFERENCE_API_KEY stripped after an interrupted run, covering provider/inference state-machine propagation and credential recovery behavior changed by this PR.
  • double-onboard-vitest (medium-high): Covers repeated onboard/recreate sandbox lifecycle, gateway reuse, and stale registry recovery. This is the closest existing live coverage for the recreate-sandbox recovery class affected by gateway credential reuse.
  • cloud-onboard-vitest (medium-high): Validates the normal live NVIDIA Endpoints cloud onboarding path with a real key, ensuring the new skip/reuse branch did not break explicit/standard build-provider validation and inference setup.

Optional E2E

  • inference-routing-vitest (medium): Useful adjacent confidence for gateway inference provider routing, credential classification, and credential isolation because this PR changes when endpoint validation and host smoke probes are run.
  • credential-migration-vitest (medium): Optional adjacent credential confidence: verifies onboarding can source provider credentials from non-env storage and register them with OpenShell, though it does not directly cover gateway-only recreate reuse.

New E2E recommendations

  • credentials/gateway-reuse (high): No existing live E2E appears to exactly cover non-interactive nemoclaw onboard --recreate-sandbox recovering the NVIDIA Endpoints build provider from an existing OpenShell gateway credential while the host has no NVIDIA_INFERENCE_API_KEY/NVIDIA_API_KEY, then verifying endpoint re-validation and host smoke are skipped but sandbox inference remains usable.
    • Suggested test: Add a focused live Vitest E2E scenario for build-provider recreate credential reuse without a local host key, promoted from or modeled after test/onboard-build-recreate-credential-reuse.test.ts.

Dispatch hint

  • Workflow: e2e-vitest-scenarios.yaml
  • jobs input: onboard-resume-vitest,double-onboard-vitest,cloud-onboard-vitest

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: ubuntu-repo-cloud-openclaw
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

  • ubuntu-repo-cloud-openclaw: The PR changes core non-interactive onboarding/provider-selection and inference setup paths for the NVIDIA Endpoints cloud OpenClaw provider, including credential reuse and host smoke-probe behavior. The live-supported Ubuntu cloud OpenClaw scenario is the smallest Vitest scenario that exercises the affected build provider onboarding and inference route surface.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional Vitest E2E scenarios

  • None.

Relevant changed files

  • src/lib/onboard.ts
  • src/lib/onboard/build-credential-reuse.ts
  • src/lib/onboard/machine/handlers/provider-inference.ts
  • src/lib/onboard/setup-nim-ollama.ts
  • src/lib/onboard/setup-nim-selection.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/onboard.ts (1)

3638-3669: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

CI blocker: src/lib/onboard.ts growth guardrail is failing and will block merge.

The new logic is correct for #5441, but CI currently fails because this file grew past the allowed net budget. Please move this new branch/flag handling into an src/lib/onboard/* helper and keep src/lib/onboard.ts net-neutral.

As per coding guidelines, src/lib/onboard.ts is core onboarding logic, and this repo enforces growth guardrails against net growth in this top-level entrypoint.

Also applies to: 3910-3916

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard.ts` around lines 3638 - 3669, The file src/lib/onboard.ts has
exceeded its growth budget due to the new NVIDIA API key validation and
reuseGatewayCredentialWithoutLocalKey flag handling logic. Extract the branch
containing the isNonInteractive() check along with the calls to
validateNvidiaApiKeyValue, providerExistsInGateway, and
logMissingNvidiaApiKeyHelp, plus the setting of the
reuseGatewayCredentialWithoutLocalKey flag, into a new helper function in the
src/lib/onboard/ directory. Replace the extracted logic in src/lib/onboard.ts
(at lines 3638-3669 and the additional location at lines 3910-3916) with calls
to this new helper function, ensuring the helper returns or sets the
reuseGatewayCredentialWithoutLocalKey value as needed. This will keep
src/lib/onboard.ts net-neutral while preserving the correct functionality for
issue `#5441`.

Sources: Coding guidelines, Pipeline failures

🧹 Nitpick comments (1)
src/lib/onboard.ts (1)

3346-3943: Run the recommended onboarding E2E slice for this change before merge.

Given this is a stage [3/8] provider-selection regression in core onboarding, run at least:
cloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2e,channels-stop-start-e2e,messaging-compatible-endpoint-e2e,bedrock-runtime-compatible-anthropic-e2e.

As per coding guidelines, these are the explicitly recommended suites for onboarding and provider-path regressions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard.ts` around lines 3346 - 3943, Before merging this pull
request, you must run the recommended E2E test suites to verify no regressions
were introduced in the setupNim function and provider-selection flow. Execute
the following E2E test slices: cloud-e2e, sandbox-operations-e2e,
rebuild-openclaw-e2e, channels-stop-start-e2e,
messaging-compatible-endpoint-e2e, and bedrock-runtime-compatible-anthropic-e2e.
These suites are required per coding guidelines for onboarding and provider-path
regressions since this change affects stage 3/8 of core provider selection in
the setupNim function.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/onboard.ts`:
- Around line 3638-3669: The file src/lib/onboard.ts has exceeded its growth
budget due to the new NVIDIA API key validation and
reuseGatewayCredentialWithoutLocalKey flag handling logic. Extract the branch
containing the isNonInteractive() check along with the calls to
validateNvidiaApiKeyValue, providerExistsInGateway, and
logMissingNvidiaApiKeyHelp, plus the setting of the
reuseGatewayCredentialWithoutLocalKey flag, into a new helper function in the
src/lib/onboard/ directory. Replace the extracted logic in src/lib/onboard.ts
(at lines 3638-3669 and the additional location at lines 3910-3916) with calls
to this new helper function, ensuring the helper returns or sets the
reuseGatewayCredentialWithoutLocalKey value as needed. This will keep
src/lib/onboard.ts net-neutral while preserving the correct functionality for
issue `#5441`.

---

Nitpick comments:
In `@src/lib/onboard.ts`:
- Around line 3346-3943: Before merging this pull request, you must run the
recommended E2E test suites to verify no regressions were introduced in the
setupNim function and provider-selection flow. Execute the following E2E test
slices: cloud-e2e, sandbox-operations-e2e, rebuild-openclaw-e2e,
channels-stop-start-e2e, messaging-compatible-endpoint-e2e, and
bedrock-runtime-compatible-anthropic-e2e. These suites are required per coding
guidelines for onboarding and provider-path regressions since this change
affects stage 3/8 of core provider selection in the setupNim function.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d29e93b0-da63-4fdf-b3dc-74e5f26604d5

📥 Commits

Reviewing files that changed from the base of the PR and between f4f3c58 and bed078a.

📒 Files selected for processing (2)
  • src/lib/onboard.ts
  • test/onboard-build-recreate-credential-reuse.test.ts

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 5 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 4 still apply, 0 new items found

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • Source-of-truth review needed: Non-interactive NVIDIA Endpoints gateway credential reuse: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: `resolveNonInteractiveBuildCredential` returns true from `providerExistsInGateway(provider)` for recovered build selections, and `setupInference` skips host smoke when the propagated flag is true.
  • Smoke-skipped gateway reuse still accepts mismatched inference routes (src/lib/onboard.ts:979): When `skipHostInferenceSmoke` is true, `setupInference` skips the host smoke probe and relies on `verifyInferenceRoute(provider, model)`. That verifier still ignores its `provider` and `model` arguments and only checks that `openshell inference get` is non-empty/not configured. Because the build provider also applies `openshell inference set --no-verify`, a stale route or a route for a different provider/model can be treated as configured after both endpoint re-validation and host smoke are skipped.
    • Recommendation: Parse `openshell inference get` for this final verification and require the live provider/model to match the selected provider/model, or use a gateway-backed verification that proves the selected route is usable. Add a negative test where `inference get` reports a different provider/model after setup and assert setup fails.
    • Evidence: `verifyInferenceRoute(_provider, _model)` ignores both parameters; `setupInference` prints `Reusing existing gateway credential; skipping host inference smoke.` when `options.skipHostInferenceSmoke === true`.
  • Gateway provider existence is still treated as credential validation (src/lib/onboard/build-credential-reuse.ts:42): The reuse branch is now limited to recovered sandbox selection, which is an improvement, but it still treats `providerExistsInGateway(provider)` as proof that the gateway holds a valid, reusable NVIDIA credential for this recovered sandbox. The provider helper documents that it is a gateway-level lookup and does not verify sandbox-scoped attachment. If the gateway provider is stale, invalid, or unrelated to the recovered sandbox, this path skips both endpoint re-validation and host smoke.
    • Recommendation: Use an authoritative source that proves the recovered route/provider credential is valid for the selected sandbox, or explicitly document the OpenShell limitation and require route provider/model matching plus stale-provider negative tests. Include a removal condition, such as when OpenShell exposes sandbox-scoped provider credential validation.
    • Evidence: `resolveNonInteractiveBuildCredential` returns `true` when `recoveredFromSandbox` and `providerExistsInGateway(provider)` are true; `src/lib/onboard/providers.ts` documents `providerExistsInGateway` as `openshell provider get` and says it does not verify sandbox-scoped attachment.
  • Full recreate and policy-preset acceptance is only partially covered (test/onboard-build-recreate-credential-reuse.test.ts:28): The linked issue's expected result requires non-interactive `onboard --recreate-sandbox` to proceed through all 8 stages and apply `NEMOCLAW_POLICY_PRESETS`. The new regression covers the proximate credential-reuse behavior through `setupNim` and `setupInference`, but it does not invoke the full CLI recreate lifecycle, sandbox creation, or policy preset stage.
    • Recommendation: Add or identify a focused full-flow/runtime test that drives non-interactive recreate far enough to verify policy preset application, or narrow the PR acceptance claim to the credential-validation stage and track full 8-stage/policy validation separately.
    • Evidence: The test imports `setupNim` and `setupInference` from `dist/lib/onboard.js`; it does not run `nemoclaw onboard --name ... --recreate-sandbox --non-interactive` or assert `NEMOCLAW_POLICY_PRESETS` behavior.
  • Regression test over-mocks OpenShell success paths (test/onboard-build-recreate-credential-reuse.test.ts:42): The fake `openshell` returns success for every command except special `inference get` and `provider get` branches. This proves host curl probes are skipped and explicit provider selection fails without a local key, but it cannot catch failures in provider update, `inference set`, stale gateway credentials, or route mismatch after the smoke probe is skipped.
    • Recommendation: Add negative tests where `provider get` succeeds but `openshell inference set` fails, and where `inference set` succeeds but `inference get` reports a different provider/model. These should assert setup fails instead of marking the route configured.
    • Evidence: The fake `openshell` script ends with `exit 0` for all unhandled commands, so critical route/provider operations always appear successful.

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — Recovered `nvidia-prod` with no host NVIDIA key runs full non-interactive `nemoclaw onboard --name <existing> --recreate-sandbox --non-interactive` far enough to assert no stage [3/8] abort and `NEMOCLAW_POLICY_PRESETS` application occurs.. The changed behavior crosses host env staging, credential storage, OpenShell gateway provider state, inference route configuration, sandbox recreation, and policy preset lifecycle boundaries. The helper-level child-process regression is useful, but it uses fake OpenShell success paths for critical operations.
  • **Runtime validation** — Gateway provider exists but `openshell inference set` fails: `setupInference` surfaces the provider configuration failure and does not mark the route configured.. The changed behavior crosses host env staging, credential storage, OpenShell gateway provider state, inference route configuration, sandbox recreation, and policy preset lifecycle boundaries. The helper-level child-process regression is useful, but it uses fake OpenShell success paths for critical operations.
  • **Runtime validation** — Gateway provider exists and `openshell inference get` reports a different provider/model after setup: final route verification fails.. The changed behavior crosses host env staging, credential storage, OpenShell gateway provider state, inference route configuration, sandbox recreation, and policy preset lifecycle boundaries. The helper-level child-process regression is useful, but it uses fake OpenShell success paths for critical operations.
  • **Runtime validation** — Recovered sandbox provider is `nvidia-prod` but gateway provider state is stale or unrelated: endpoint and host smoke are not both skipped unless the live route provider/model matches the recovered selection.. The changed behavior crosses host env staging, credential storage, OpenShell gateway provider state, inference route configuration, sandbox recreation, and policy preset lifecycle boundaries. The helper-level child-process regression is useful, but it uses fake OpenShell success paths for critical operations.
  • **Runtime validation** — Recovered build credential reuse keeps explicit-provider behavior: `NEMOCLAW_PROVIDER=build` without a local key fails before curl even if the gateway provider exists.. The changed behavior crosses host env staging, credential storage, OpenShell gateway provider state, inference route configuration, sandbox recreation, and policy preset lifecycle boundaries. The helper-level child-process regression is useful, but it uses fake OpenShell success paths for critical operations.
  • **Regression test over-mocks OpenShell success paths** — Add negative tests where `provider get` succeeds but `openshell inference set` fails, and where `inference set` succeeds but `inference get` reports a different provider/model. These should assert setup fails instead of marking the route configured.
  • **Acceptance clause:** [DGX Spark][Onboard] nemoclaw onboard --recreate-sandbox non-interactive aborts at [3/8] with "NVIDIA Endpoints endpoint validation failed" — add test evidence or identify existing coverage. The diff skips endpoint re-validation and host smoke for recovered build selections with a gateway credential; the test covers `setupNim` plus `setupInference`, not the full CLI recreate command.
  • **Acceptance clause:** On DGX Spark with NemoClaw v0.0.64, running `nemoclaw onboard --recreate-sandbox` in non-interactive mode fails at [3/8] (Configuring inference provider) with "NVIDIA Endpoints endpoint validation failed. Validation details were omitted to avoid exposing credentials." The same API key succeeds in interactive onboard within the same shell session. The non-interactive flow recovers the provider from saved sandbox build state and then fails credential validation. — add test evidence or identify existing coverage. `resolveBuildPreferredInferenceApi` avoids the unauthenticated endpoint probe in the recovered gateway-credential path, and the test asserts no fake curl endpoint probe occurs. No DGX Spark/aarch64/full-command runtime evidence is in the diff.
Since last review details

Current findings:

  • Source-of-truth review needed: Non-interactive NVIDIA Endpoints gateway credential reuse: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: `resolveNonInteractiveBuildCredential` returns true from `providerExistsInGateway(provider)` for recovered build selections, and `setupInference` skips host smoke when the propagated flag is true.
  • Smoke-skipped gateway reuse still accepts mismatched inference routes (src/lib/onboard.ts:979): When `skipHostInferenceSmoke` is true, `setupInference` skips the host smoke probe and relies on `verifyInferenceRoute(provider, model)`. That verifier still ignores its `provider` and `model` arguments and only checks that `openshell inference get` is non-empty/not configured. Because the build provider also applies `openshell inference set --no-verify`, a stale route or a route for a different provider/model can be treated as configured after both endpoint re-validation and host smoke are skipped.
    • Recommendation: Parse `openshell inference get` for this final verification and require the live provider/model to match the selected provider/model, or use a gateway-backed verification that proves the selected route is usable. Add a negative test where `inference get` reports a different provider/model after setup and assert setup fails.
    • Evidence: `verifyInferenceRoute(_provider, _model)` ignores both parameters; `setupInference` prints `Reusing existing gateway credential; skipping host inference smoke.` when `options.skipHostInferenceSmoke === true`.
  • Gateway provider existence is still treated as credential validation (src/lib/onboard/build-credential-reuse.ts:42): The reuse branch is now limited to recovered sandbox selection, which is an improvement, but it still treats `providerExistsInGateway(provider)` as proof that the gateway holds a valid, reusable NVIDIA credential for this recovered sandbox. The provider helper documents that it is a gateway-level lookup and does not verify sandbox-scoped attachment. If the gateway provider is stale, invalid, or unrelated to the recovered sandbox, this path skips both endpoint re-validation and host smoke.
    • Recommendation: Use an authoritative source that proves the recovered route/provider credential is valid for the selected sandbox, or explicitly document the OpenShell limitation and require route provider/model matching plus stale-provider negative tests. Include a removal condition, such as when OpenShell exposes sandbox-scoped provider credential validation.
    • Evidence: `resolveNonInteractiveBuildCredential` returns `true` when `recoveredFromSandbox` and `providerExistsInGateway(provider)` are true; `src/lib/onboard/providers.ts` documents `providerExistsInGateway` as `openshell provider get` and says it does not verify sandbox-scoped attachment.
  • Full recreate and policy-preset acceptance is only partially covered (test/onboard-build-recreate-credential-reuse.test.ts:28): The linked issue's expected result requires non-interactive `onboard --recreate-sandbox` to proceed through all 8 stages and apply `NEMOCLAW_POLICY_PRESETS`. The new regression covers the proximate credential-reuse behavior through `setupNim` and `setupInference`, but it does not invoke the full CLI recreate lifecycle, sandbox creation, or policy preset stage.
    • Recommendation: Add or identify a focused full-flow/runtime test that drives non-interactive recreate far enough to verify policy preset application, or narrow the PR acceptance claim to the credential-validation stage and track full 8-stage/policy validation separately.
    • Evidence: The test imports `setupNim` and `setupInference` from `dist/lib/onboard.js`; it does not run `nemoclaw onboard --name ... --recreate-sandbox --non-interactive` or assert `NEMOCLAW_POLICY_PRESETS` behavior.
  • Regression test over-mocks OpenShell success paths (test/onboard-build-recreate-credential-reuse.test.ts:42): The fake `openshell` returns success for every command except special `inference get` and `provider get` branches. This proves host curl probes are skipped and explicit provider selection fails without a local key, but it cannot catch failures in provider update, `inference set`, stale gateway credentials, or route mismatch after the smoke probe is skipped.
    • Recommendation: Add negative tests where `provider get` succeeds but `openshell inference set` fails, and where `inference set` succeeds but `inference get` reports a different provider/model. These should assert setup fails instead of marking the route configured.
    • Evidence: The fake `openshell` script ends with `exit 0` for all unhandled commands, so critical route/provider operations always appear successful.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

The #5441 fix added net lines to src/lib/onboard.ts, tripping the codebase
growth guardrail (the top-level onboard entrypoint must be net-neutral or
smaller; growth is allowed under src/lib/onboard/).

Move the non-interactive NVIDIA Endpoints ('build') credential gate and the
endpoint-validation/skip decision into src/lib/onboard/build-credential-reuse.ts
(resolveNonInteractiveBuildCredential + resolveBuildPreferredInferenceApi).
Behavior is unchanged: when the gateway already holds a validated credential
and no local key is staged, reuse it and skip endpoint re-validation. The
onboard.ts call sites shrink, leaving the file net-smaller than main.

Capture the flow-narrowed model into a const so the deferred probe closure
keeps the string narrowing the original synchronous call relied on.

Refs #5441

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cv cv changed the title fix: address issue #5441 fix: reuse gateway NVIDIA credential for non-interactive recreate Jun 17, 2026
@cv

cv commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Addressed the merge conflict and review feedback:

  • Merged current main into the PR branch and resolved the setupNim refactor conflict.
  • Kept NVIDIA Endpoints gateway-credential reuse scoped to recovered sandbox/non-interactive recreate flows. Explicit non-interactive build selections still require a local NVIDIA key.
  • Carried the gateway-credential reuse decision through provider inference and skipped the host-side direct smoke probe only for that recovered gateway-credential case.
  • Extended the regression test to run setupNim + setupInference without a local NVIDIA key and added a negative test for explicit build selection.
  • Updated the PR title to describe the fix.

@cv

cv commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Correction to previous comment: the net-negative file is src/lib/onboard.ts (+39/-40 in the PR file diff).

@cv
cv merged commit 3467cbe into main Jun 17, 2026
37 checks passed
@cv
cv deleted the auto/fix-5441-dgx-spark-onboard-nemoclaw-onboard-recreate branch June 17, 2026 22:02
@wscurran wscurran added NV QA Bugs found by the NVIDIA QA Team UAT Issues flagged for User Acceptance Testing. labels Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

NV QA Bugs found by the NVIDIA QA Team UAT Issues flagged for User Acceptance Testing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DGX Spark][Onboard] nemoclaw onboard --recreate-sandbox non-interactive aborts at [3/8] with "NVIDIA Endpoints endpoint validation failed"

3 participants