fix(onboard): handle Docker Desktop WSL CDI injection failures - #5198
Conversation
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR addresses GPU CDI injection failures on Docker Desktop WSL by detecting the specific failure pattern, refusing the patch opt-out on Docker Desktop WSL to keep the compatibility path enabled, and emitting targeted recovery guidance pointing to ChangesGPU CDI injection detection and Docker Desktop WSL handling
🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
🌿 Preview your docs: https://nvidia-preview-pr-5198.docs.buildwithfern.com/nemoclaw |
E2E Advisor RecommendationRequired E2E: None Full advisor summaryE2E Recommendation AdvisorFailed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-advisor-raw-output.txt |
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: None Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorFailed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-scenario-advisor-raw-output.txt |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/onboard.ts`:
- Around line 3467-3473: The new docker-desktop detection and wiring
(dockerDesktopWsl = preflightUtils.detectWslDockerDesktopStatus({}) ===
"docker-desktop"; and the call to
dockerGpuSandboxCreate.resolveDockerGpuSandboxCreatePlan(...)) must be moved so
the file stays net-neutral: relocate the detectWslDockerDesktopStatus call and
the dockerDesktopWsl variable into an existing helper boundary (for example
inside the helper that already computes sandbox GPU config or inside
dockerGpuSandboxCreate.resolveDockerGpuSandboxCreatePlan) so the top-level hunk
no longer grows the file; update resolveDockerGpuSandboxCreatePlan invocations
to accept or compute dockerDesktopWsl internally (using
isLinuxDockerDriverGatewayEnabled and effectiveSandboxGpuConfig as before), or
remove an equivalent nearby helper/unused logging line to offset +3 lines if
moving is not feasible. Ensure references to detectWslDockerDesktopStatus,
dockerDesktopWsl, dockerGpuSandboxCreate.resolveDockerGpuSandboxCreatePlan,
effectiveSandboxGpuConfig, and isLinuxDockerDriverGatewayEnabled are updated
accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bb932bc6-5fa2-4ce9-a5a6-44e14e7b4985
📒 Files selected for processing (10)
docs/reference/troubleshooting.mdxsrc/lib/build-context.test.tssrc/lib/build-context.tssrc/lib/onboard.tssrc/lib/onboard/docker-gpu-patch.test.tssrc/lib/onboard/docker-gpu-patch.tssrc/lib/onboard/docker-gpu-sandbox-create.tssrc/lib/onboard/preflight.tssrc/lib/validation.test.tssrc/lib/validation.ts
PR Review AdvisorFindings: 0 needs attention, 3 worth checking, 0 nice ideas Review findings🛠️ Needs attention
🔎 Worth checking
🌱 Nice ideas
Consider writing more tests for
Since last review detailsCurrent findings:
This is an automated advisory review. A human maintainer must make the final merge decision. |
Selective E2E Results — ❌ Some jobs failedRun: 27320066526
|
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Selective E2E Results —
|
| Job | Result |
|---|---|
| gpu-e2e | ⏭️ skipped |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/onboard/command-support.ts`:
- Around line 69-70: Update the help text for the --no-sandbox-gpu option to
remove the implication that it must be paired with --no-gpu; clarify that
--no-sandbox-gpu is equivalent to setting NEMOCLAW_SANDBOX_GPU=0 and can be used
as an alternative to --no-gpu (not combined), and ensure consistency with the
gpu_cdi_injection_failed guidance which suggests using either --no-gpu or
NEMOCLAW_SANDBOX_GPU=0 (and the recovery command onboard --resume --no-gpu).
Reference the --no-sandbox-gpu option and the NEMOCLAW_SANDBOX_GPU variable in
the description string in command-support.ts and make the wording match the
guidance in build-context.ts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1084cc0d-232d-441a-9393-437a58b18886
📒 Files selected for processing (7)
src/lib/build-context.tssrc/lib/onboard/command-support.tssrc/lib/onboard/docker-gpu-local-inference.tssrc/lib/onboard/docker-gpu-patch-wsl.test.tssrc/lib/onboard/docker-gpu-patch.tssrc/lib/onboard/docker-gpu-sandbox-create.tssrc/lib/validation.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- src/lib/build-context.ts
- src/lib/validation.test.ts
- src/lib/onboard/docker-gpu-patch.ts
…tection Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Selective E2E Results —
|
| Job | Result |
|---|---|
| gpu-e2e | ⏭️ skipped |
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
## Summary Refreshes release-prep documentation for NemoClaw v0.0.65. Adds the v0.0.65 release-notes section and refreshes generated `nemoclaw-user-*` skills from the Fern MDX source docs. ## Changes - Added the v0.0.65 release notes to `docs/about/release-notes.mdx` with links to the deeper docs pages for lifecycle, troubleshooting, inference, CLI commands, messaging, credentials, network policy, Hermes, and sub-agents. - Regenerated the `nemoclaw-user-*` skills with `scripts/docs-to-skills.py` so release-prep skill output matches the merged source docs. - Used the v0.0.65 announcement discussion as release context: #5472. ## Source Summary - #2492 -> `docs/about/release-notes.mdx`: Documents deadline-based gateway wait reliability in the v0.0.65 recovery summary. - #4958 -> `docs/about/release-notes.mdx`: Documents re-execed OpenClaw gateway health check recovery in the sandbox recovery summary. - #5163 -> `docs/about/release-notes.mdx`: Documents safer uninstall TTY confirmation behavior in the day-two CLI summary. - #5178 -> `docs/about/release-notes.mdx`: Documents fail-closed config restore merge behavior in the rebuild and restore summary. - #5179 -> `docs/about/release-notes.mdx`: Documents WeChat QR token redaction in the messaging summary. - #5182 -> `docs/about/release-notes.mdx`: Documents sustained gateway serving checks in the recovery summary. - #5194 -> `docs/about/release-notes.mdx`: Documents model-router teardown during uninstall in the day-two CLI summary. - #5195 -> `docs/about/release-notes.mdx`: Documents Shields auto-restore lock reconfirmation in the rebuild and restore summary. - #5198 -> `docs/about/release-notes.mdx`: Documents Docker Desktop WSL CDI injection failure handling in the onboarding diagnostics summary. - #5201 -> `docs/about/release-notes.mdx`: Documents sandbox download/upload wrappers and sessions export in the day-two CLI summary. - #5205 -> `docs/about/release-notes.mdx`: Documents reporter-owned model metadata preservation in the rebuild and restore summary. - #5214 -> `docs/about/release-notes.mdx`: Documents managed vLLM model preflight before side effects in the inference setup summary. - #5215 -> `docs/about/release-notes.mdx`: Documents managed vLLM extra serve arguments in the inference setup summary. - #5216 -> `docs/about/release-notes.mdx`: Documents silent OpenClaw runtime fallback surfacing in the onboarding diagnostics summary. - #5225 -> `docs/about/release-notes.mdx`: Documents persisted sandbox gateway lookup in the gateway recovery summary. - #5238 -> `docs/about/release-notes.mdx`: Documents sub-agent gateway dial-back through the sandbox interface in the Hermes and sub-agent summary. - #5248 -> `docs/about/release-notes.mdx`: Documents Discord per-account proxy resolution in the messaging summary. - #5264 -> `docs/about/release-notes.mdx`: Documents reserved Hermes port `8642` handling in the Hermes compatibility summary. - #5267 -> `docs/about/release-notes.mdx`: Documents the narrower Hermes baseline policy in the Hermes compatibility summary. - #5321 -> `docs/about/release-notes.mdx`: Documents restored gateway guard chains in the gateway recovery summary. - #5328 -> `docs/about/release-notes.mdx`: Documents compact persisted messaging plans in the messaging summary. - #5338 -> `docs/about/release-notes.mdx`: Documents manifest channel migration in the messaging summary. - #5352 -> `docs/about/release-notes.mdx`: Documents persisted agent preservation through registry recovery in the rebuild and restore summary. - #5371 -> `.agents/skills/nemoclaw-user-reference/references/commands.md`: Refreshes generated skill output for custom build cache and layer-ordering source docs. - #5379 -> `docs/about/release-notes.mdx`: Documents dashboard port allocation across multiple NemoClaw gateways in the recovery summary. - #5382 -> `docs/about/release-notes.mdx`: Documents recovery when an active gateway has no sandbox spec in the recovery summary. - #5389 -> `.agents/skills/nemoclaw-user-reference/references/troubleshooting.md`: Refreshes generated skill output for declared agent `forward_ports` recovery source docs. - #5400 -> `docs/about/release-notes.mdx`: Documents bounded compatible endpoint probes in the inference setup summary. - #5410 -> `docs/about/release-notes.mdx`: Documents provider credential hash removal from sandbox registry entries in the messaging summary. - #5418 -> `docs/about/release-notes.mdx`: Documents summarized inference validation failures in the onboarding diagnostics summary. - #5457 -> `docs/about/release-notes.mdx`: Documents context-window recomputation after runtime model switches in the inference setup summary. - #5463 -> `docs/about/release-notes.mdx`: Documents cleanup of hard-coded messaging channel stragglers in the messaging summary. ## Skipped - #5366 matched `docs/.docs-skip` entries through skipped experimental paths, so this PR does not add new release-note text for that commit. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [x] Doc only (includes code sample changes) ## Verification - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [ ] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [ ] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Verification notes: - `npm run docs` passed after rerunning outside the sandbox. Fern reported 0 errors and 1 hidden warning. - The first sandboxed `npm run docs` attempt failed before validation because `tsx` could not create its local IPC pipe under sandbox restrictions. - `npm run build:cli` passed before push to refresh the local `dist/` artifacts used by the CLI typecheck hook. - `npm test` was not run because this is a docs-only release refresh. --- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Released NemoClaw v0.0.65 with improved gateway/sandbox recovery, safer day-two workflows, and enhanced Hermes compatibility. * Added managed vLLM extra-arguments configuration via `NEMOCLAW_VLLM_EXTRA_ARGS_JSON`. * Added Hermes troubleshooting guidance for port forwarding and health checks. * **Documentation** * Updated NVIDIA Endpoints/NIM setup and examples to use `NVIDIA_INFERENCE_API_KEY`. * Refined NVIDIA network policy and Model Router API base configuration. * Expanded CLI/environment variable documentation (including sub-agent gateway connectivity) and plugin build performance tips. * **Tests** * Expanded Vitest-backed E2E release validation coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
On Docker Desktop + WSL2 with an NVIDIA GPU, onboard's `[6/8]` Docker
GPU patch recreates the sandbox container with `--device
nvidia.com/gpu=all` (CDI syntax) and fails:
```
CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all
```
even though preflight already logs that it will use the `--gpus`
compatibility path. The only workaround today is `--no-gpu` /
`NEMOCLAW_SANDBOX_GPU=0`, which disables GPU entirely.
## Root cause
Docker Desktop advertises CDI spec **directories**, so
`dockerReportsNvidiaCdiDevices()` returns true and
`buildDockerGpuModeCandidates()` offers CDI as the first candidate. The
create-only probe (`docker create … true`) passes, but the real recreate
fails because the WSL distro exposes **no usable `nvidia.com/gpu`
spec**. The Docker Desktop WSL status was detected at preflight but
never reached the mode selector — `selectDockerGpuPatchMode` only
received `{image, device, backend}`.
PR #5198 (which closed #5180) added the CDI-injection failure
classification, the `--no-gpu` recovery hint, and the warning that
`NEMOCLAW_DOCKER_GPU_PATCH=0` is ignored on this runtime — but it did
not change mode selection. This is the unaddressed root cause.
## Fix
Thread the existing Docker Desktop WSL detection
(`isDockerDesktopWslRuntime()`, already used to gate the patch) through
`selectDockerGpuPatchMode` into `buildDockerGpuModeCandidates`, and skip
the CDI candidate when on Docker Desktop WSL so the patch uses `--gpus
all` — the path preflight already commits to.
- Native Docker-CDI hosts are **unaffected**: they still prefer CDI,
preserving the gateway supervisor-wiring contract from #4948.
- The flag is resolved via the cached detector in
`docker-gpu-sandbox-create.ts`, so no change to `onboard.ts` and no
extra `docker info` calls.
## Testing
- New unit tests in `docker-gpu-patch-wsl.test.ts`: CDI is skipped
(first candidate is `--gpus all`) when `dockerDesktopWsl` is true even
with CDI advertised, and CDI is still preferred otherwise.
- `tsc -p tsconfig.src.json` clean; GPU-patch suites pass (remaining
failures are pre-existing Windows-only `/etc/cdi` path tests, identical
on `main`).
## Notes / follow-up
- This step is only reached after the `[2/8]` gateway-bind issue (#5513,
fix in #5534).
- Separate latent bug still open: on an **early** patch failure the
original sandbox is already renamed to
`*-nemoclaw-gpu-backup-<timestamp>` before container creation, and only
the new container is removed — leaving an orphan backup. Happy to follow
up with a focused PR for that cleanup.
Fixes #5512.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced GPU configuration for Docker Desktop on Windows Subsystem for
Linux (WSL). The system now properly detects WSL runtime environments
and automatically selects GPU acceleration modes that work reliably on
Docker Desktop WSL, avoiding GPU modes that may not be available or
incompatible within that specific environment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
…VIDIA#5537) ## Summary On Docker Desktop + WSL2 with an NVIDIA GPU, onboard's `[6/8]` Docker GPU patch recreates the sandbox container with `--device nvidia.com/gpu=all` (CDI syntax) and fails: ``` CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all ``` even though preflight already logs that it will use the `--gpus` compatibility path. The only workaround today is `--no-gpu` / `NEMOCLAW_SANDBOX_GPU=0`, which disables GPU entirely. ## Root cause Docker Desktop advertises CDI spec **directories**, so `dockerReportsNvidiaCdiDevices()` returns true and `buildDockerGpuModeCandidates()` offers CDI as the first candidate. The create-only probe (`docker create … true`) passes, but the real recreate fails because the WSL distro exposes **no usable `nvidia.com/gpu` spec**. The Docker Desktop WSL status was detected at preflight but never reached the mode selector — `selectDockerGpuPatchMode` only received `{image, device, backend}`. PR NVIDIA#5198 (which closed NVIDIA#5180) added the CDI-injection failure classification, the `--no-gpu` recovery hint, and the warning that `NEMOCLAW_DOCKER_GPU_PATCH=0` is ignored on this runtime — but it did not change mode selection. This is the unaddressed root cause. ## Fix Thread the existing Docker Desktop WSL detection (`isDockerDesktopWslRuntime()`, already used to gate the patch) through `selectDockerGpuPatchMode` into `buildDockerGpuModeCandidates`, and skip the CDI candidate when on Docker Desktop WSL so the patch uses `--gpus all` — the path preflight already commits to. - Native Docker-CDI hosts are **unaffected**: they still prefer CDI, preserving the gateway supervisor-wiring contract from NVIDIA#4948. - The flag is resolved via the cached detector in `docker-gpu-sandbox-create.ts`, so no change to `onboard.ts` and no extra `docker info` calls. ## Testing - New unit tests in `docker-gpu-patch-wsl.test.ts`: CDI is skipped (first candidate is `--gpus all`) when `dockerDesktopWsl` is true even with CDI advertised, and CDI is still preferred otherwise. - `tsc -p tsconfig.src.json` clean; GPU-patch suites pass (remaining failures are pre-existing Windows-only `/etc/cdi` path tests, identical on `main`). ## Notes / follow-up - This step is only reached after the `[2/8]` gateway-bind issue (NVIDIA#5513, fix in NVIDIA#5534). - Separate latent bug still open: on an **early** patch failure the original sandbox is already renamed to `*-nemoclaw-gpu-backup-<timestamp>` before container creation, and only the new container is removed — leaving an orphan backup. Happy to follow up with a focused PR for that cleanup. Fixes NVIDIA#5512. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Enhanced GPU configuration for Docker Desktop on Windows Subsystem for Linux (WSL). The system now properly detects WSL runtime environments and automatically selects GPU acceleration modes that work reliably on Docker Desktop WSL, avoiding GPU modes that may not be available or incompatible within that specific environment. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com> Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Summary
On Docker Desktop WSL the OpenShell gateway issues
docker create --device nvidia.com/gpu=allindependently of NemoClaw's own GPU patch, so settingNEMOCLAW_DOCKER_GPU_PATCH=0shifts the failure to a less actionable CDI injection error. Classify the new failure mode, point the recovery hint at--no-gpu/NEMOCLAW_SANDBOX_GPU=0, and refuse the env opt-out on Docker Desktop WSL where the patch is required.Related Issue
Fixes #5180
Changes
src/lib/validation.ts: newgpu_cdi_injection_failedfailure kind + classifier arm matchingCDI device injection failed/unresolvable CDI deviceswhen paired withnvidia.com/gpu. Ordered ahead of the genericCreated sandbox:arm so mixed output keeps GPU-specific guidance.src/lib/build-context.ts: recovery hint for the new kind explains the gateway issues the injection itself, calls out thatNEMOCLAW_DOCKER_GPU_PATCH=0does not bypass it, and points at--no-gpu/NEMOCLAW_SANDBOX_GPU=0plusonboard --resume --no-gpu.src/lib/onboard/docker-gpu-patch.ts:shouldApplyDockerGpuPatchaccepts an optionaldockerDesktopWslflag and refuses the env opt-out on that runtime with a one-line warning; the patch-failure escape-hatch block now lists bothNEMOCLAW_DOCKER_GPU_PATCH=0andNEMOCLAW_SANDBOX_GPU=0with scope notes.src/lib/onboard/docker-gpu-sandbox-create.ts: detects Docker Desktop WSL insideresolveDockerGpuSandboxCreatePlan(cached viaisDockerDesktopWslRuntime) so create-plan and post-ready local-inference paths share the same runtime decision without growingsrc/lib/onboard.ts.src/lib/onboard/docker-gpu-local-inference.ts: threads the resolveddockerDesktopWsl(or falls back to the shared cache) intoshouldUseDockerGpuPatchHostNetworkandverifyDockerGpuSandboxLocalInference, so the host-network downgrade and post-ready probe still run when the opt-out is ignored.src/lib/onboard/command-support.ts:--no-sandbox-gpudescription notes theNEMOCLAW_SANDBOX_GPU=0equivalence and that it is the alternative to--no-gpuon Docker Desktop WSL.docs/reference/troubleshooting.mdx: Docker Desktop WSL guidance plus a recovery entry forCDI device injection failed.docker-gpu-patch-wsl.test.ts), full-path create-plan +buildSandboxGpuCreateArgs(3 cases).Follow-up not bundled: #3213 (unify warnings / advisories / fatal exits) — the new recovery hint will fold into that registry when it lands.
Type of Change
Verification
npx prek run --all-filespassesnpm testpassesnpm run docsbuilds without warnings (doc changes only)Signed-off-by: Tinson Lai tinsonl@nvidia.com
Summary by CodeRabbit
Release Notes
Documentation
New Features
Bug Fixes