Skip to content

fix(onboard): skip CDI GPU mode on Docker Desktop WSL (#5512) - #5537

Merged
jyaunches merged 1 commit into
NVIDIA:mainfrom
abhi-0906:fix/wsl-docker-desktop-gpu-patch-mode
Jun 25, 2026
Merged

fix(onboard): skip CDI GPU mode on Docker Desktop WSL (#5512)#5537
jyaunches merged 1 commit into
NVIDIA:mainfrom
abhi-0906:fix/wsl-docker-desktop-gpu-patch-mode

Conversation

@abhi-0906

@abhi-0906 abhi-0906 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

On Docker Desktop + WSL2 with an NVIDIA GPU, onboard's [6/8] Docker GPU patch recreates the sandbox container with --device nvidia.com/gpu=all (CDI syntax) and fails:

CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all

even though preflight already logs that it will use the --gpus compatibility path. The only workaround today is --no-gpu / NEMOCLAW_SANDBOX_GPU=0, which disables GPU entirely.

Root cause

Docker Desktop advertises CDI spec directories, so dockerReportsNvidiaCdiDevices() returns true and buildDockerGpuModeCandidates() offers CDI as the first candidate. The create-only probe (docker create … true) passes, but the real recreate fails because the WSL distro exposes no usable nvidia.com/gpu spec. The Docker Desktop WSL status was detected at preflight but never reached the mode selector — selectDockerGpuPatchMode only received {image, device, backend}.

PR #5198 (which closed #5180) added the CDI-injection failure classification, the --no-gpu recovery hint, and the warning that NEMOCLAW_DOCKER_GPU_PATCH=0 is ignored on this runtime — but it did not change mode selection. This is the unaddressed root cause.

Fix

Thread the existing Docker Desktop WSL detection (isDockerDesktopWslRuntime(), already used to gate the patch) through selectDockerGpuPatchMode into buildDockerGpuModeCandidates, and skip the CDI candidate when on Docker Desktop WSL so the patch uses --gpus all — the path preflight already commits to.

Testing

  • New unit tests in docker-gpu-patch-wsl.test.ts: CDI is skipped (first candidate is --gpus all) when dockerDesktopWsl is true even with CDI advertised, and CDI is still preferred otherwise.
  • tsc -p tsconfig.src.json clean; GPU-patch suites pass (remaining failures are pre-existing Windows-only /etc/cdi path tests, identical on main).

Notes / follow-up

Fixes #5512.

Summary by CodeRabbit

  • Bug Fixes
    • Enhanced GPU configuration for Docker Desktop on Windows Subsystem for Linux (WSL). The system now properly detects WSL runtime environments and automatically selects GPU acceleration modes that work reliably on Docker Desktop WSL, avoiding GPU modes that may not be available or incompatible within that specific environment.

Signed-off-by: Abhimanyu Kumar abhimanyukumar7290@gmail.com

@copy-pr-bot

copy-pr-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cv

cv commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

@abhi-0906 can you add a DCO 'Signed-off-by' to the PR description, please?

@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a dockerDesktopWsl boolean flag to GPU mode candidate selection to suppress CDI candidates on Docker Desktop WSL runtimes where CDI is advertised but unusable. The flag is threaded through four function signatures in docker-gpu-patch.ts, sourced from isDockerDesktopWslRuntime() in docker-gpu-sandbox-create.ts, and validated by new tests.

Changes

Docker Desktop WSL CDI skip in GPU patch flow

Layer / File(s) Summary
CDI skip logic and propagation chain
src/lib/onboard/docker-gpu-patch.ts
Expands buildDockerGpuModeCandidates options type with dockerDesktopWsl?: boolean and adds a guard that omits CDI from the candidate list when the flag is true. Threads the flag through selectDockerGpuPatchMode, recreateOpenShellDockerSandboxWithGpu, and applyDockerGpuPatchOrExit.
Runtime probe wiring in sandbox-create
src/lib/onboard/docker-gpu-sandbox-create.ts
Adds dockerDesktopWsl?: boolean to DockerGpuSandboxCreatePatchOptions (defaulting to isDockerDesktopWslRuntime()) and passes it into applyOptions.
Tests for WSL CDI skip behavior
src/lib/onboard/docker-gpu-patch-wsl.test.ts
Imports buildDockerGpuModeCandidates and adds a describe block asserting CDI is excluded when dockerDesktopWsl is true and that CDI remains first for non-Docker-Desktop-WSL hosts that advertise CDI.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5198: Modifies docker-gpu-patch.ts to adjust CDI-related GPU patch eligibility and messaging for Docker Desktop WSL, directly overlapping with the CDI candidate skip and dockerDesktopWsl threading introduced in this PR.

Suggested labels

bug-fix, platform: wsl, area: sandbox, v0.0.65

Suggested reviewers

  • cv

Poem

🐇 On WSL the CDI spec hides away,
So --gpus all shall carry the day.
The flag hops through each function in line,
Skipping the path that would fail every time.
No orphan containers, no exit 1 cry —
The GPU patch lands and bunnies say "hi!" 🎉

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and specifically describes the main change: skipping CDI GPU mode selection on Docker Desktop WSL to fix the onboard process failure.
Linked Issues check ✅ Passed The PR implementation threads the dockerDesktopWsl flag through the GPU patch selection logic to skip CDI mode on Docker Desktop WSL, directly addressing both issue requirements to use --gpus compatibility path instead of CDI.
Out of Scope Changes check ✅ Passed All changes are tightly scoped to GPU mode candidate selection and parameter threading; no unrelated refactoring, documentation changes, or error classification improvements are included.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@abhi-0906

Copy link
Copy Markdown
Contributor Author

Thanks @cv — added the Signed-off-by line to the PR description. I've also added it to the descriptions of the related PRs in this chain (#5534, #5536, #5541) so they're DCO-ready for squash-merge. All commits are individually signed off too; let me know if anything else is needed.

@prekshivyas prekshivyas self-assigned this Jun 22, 2026

@prekshivyas prekshivyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct and well-scoped. The `!dockerDesktopWsl` guard threads cleanly through the entire call chain, and the production path correctly defaults to `isDockerDesktopWslRuntime()` at `createDockerGpuSandboxCreatePatch`. Tests cover both skip-on-WSL and CDI-preferred-on-non-WSL paths directly against the exported function.

Minor gap: no test verifies that `createDockerGpuSandboxCreatePatch` picks up the `isDockerDesktopWslRuntime()` default when `dockerDesktopWsl` is omitted from options. Not a blocker.

abhi-0906 added a commit to abhi-0906/NemoClaw that referenced this pull request Jun 22, 2026
…NVIDIA#5512)

When the Docker GPU patch recreate `docker run` fails after the original
sandbox was already renamed to `*-nemoclaw-gpu-backup-<timestamp>`, the
early-failure path removed only the failed new container and left the
backup orphaned — stranding the sandbox with no live original and
colliding on the next retry.

Reuse the existing rollback primitive on this path: remove the failed new
container, rename the backup back to the original name, and start it, so
onboarding restores the pre-patch sandbox instead of leaking a backup
container. Adds rollbackDockerGpuPatchOnRecreateFailure to the finalize
module (resolving the real docker start/rename defaults) and records
context.rolledBack for diagnostics.

Follow-up to NVIDIA#5537 (which makes the patch succeed on Docker Desktop WSL,
so this path is no longer hit there) addressing the orphan-backup symptom
noted in NVIDIA#5512.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
@cv cv added v0.0.67 and removed v0.0.66 labels Jun 23, 2026
@wscurran wscurran added area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression platform: wsl Affects Windows Subsystem for Linux labels Jun 23, 2026
@wscurran

Copy link
Copy Markdown
Contributor

✨ Thanks for the proposed fix addressing the CDI GPU mode failure on Docker Desktop WSL2 at onboard step [6/8]. This proposes a way to skip CDI mode when the WSL distro exposes no usable nvidia.com/gpu spec, allowing the --gpus compatibility path to be used instead.


Related open PRs:


Related open issues:

@jyaunches jyaunches added v0.0.68 and removed v0.0.67 labels Jun 24, 2026
On Docker Desktop + WSL2, onboard's [6/8] Docker GPU patch recreates the
sandbox with `--device nvidia.com/gpu=all` (CDI) and fails with "CDI
device injection failed: unresolvable CDI devices nvidia.com/gpu=all",
even though preflight already commits to the `--gpus` compatibility path.
Docker Desktop advertises CDI spec directories, so dockerReportsNvidiaCdiDevices()
returns true and buildDockerGpuModeCandidates offers CDI first; the
create-only probe passes but the real recreate fails because the WSL
distro exposes no usable nvidia.com/gpu spec.

Thread the existing Docker Desktop WSL detection (isDockerDesktopWslRuntime,
already used to gate the patch) through selectDockerGpuPatchMode into
buildDockerGpuModeCandidates, and skip the CDI candidate when on Docker
Desktop WSL so the patch uses `--gpus all`. Native Docker-CDI hosts are
unaffected and still prefer CDI (preserving the NVIDIA#4948 gateway
supervisor-wiring contract).

Reached only after the [2/8] gateway-bind issue (NVIDIA#5513 / NVIDIA#5534). A
follow-up is still needed for the orphaned `*-nemoclaw-gpu-backup-*`
container left behind on an early patch failure.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
abhi-0906 added a commit to abhi-0906/NemoClaw that referenced this pull request Jun 24, 2026
…NVIDIA#5512)

When the Docker GPU patch recreate `docker run` fails after the original
sandbox was already renamed to `*-nemoclaw-gpu-backup-<timestamp>`, the
early-failure path removed only the failed new container and left the
backup orphaned — stranding the sandbox with no live original and
colliding on the next retry.

Reuse the existing rollback primitive on this path: remove the failed new
container, rename the backup back to the original name, and start it, so
onboarding restores the pre-patch sandbox instead of leaking a backup
container. Adds rollbackDockerGpuPatchOnRecreateFailure to the finalize
module (resolving the real docker start/rename defaults) and records
context.rolledBack for diagnostics.

Follow-up to NVIDIA#5537 (which makes the patch succeed on Docker Desktop WSL,
so this path is no longer hit there) addressing the orphan-backup symptom
noted in NVIDIA#5512.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
@abhi-0906
abhi-0906 force-pushed the fix/wsl-docker-desktop-gpu-patch-mode branch from ad6ffa2 to c09ffa6 Compare June 24, 2026 20:01
@jyaunches jyaunches added v0.0.69 and removed v0.0.68 labels Jun 25, 2026
jyaunches pushed a commit to abhi-0906/NemoClaw that referenced this pull request Jun 25, 2026
…NVIDIA#5512)

When the Docker GPU patch recreate `docker run` fails after the original
sandbox was already renamed to `*-nemoclaw-gpu-backup-<timestamp>`, the
early-failure path removed only the failed new container and left the
backup orphaned — stranding the sandbox with no live original and
colliding on the next retry.

Reuse the existing rollback primitive on this path: remove the failed new
container, rename the backup back to the original name, and start it, so
onboarding restores the pre-patch sandbox instead of leaking a backup
container. Adds rollbackDockerGpuPatchOnRecreateFailure to the finalize
module (resolving the real docker start/rename defaults) and records
context.rolledBack for diagnostics.

Follow-up to NVIDIA#5537 (which makes the patch succeed on Docker Desktop WSL,
so this path is no longer hit there) addressing the orphan-backup symptom
noted in NVIDIA#5512.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
@jyaunches
jyaunches merged commit 1af474b into NVIDIA:main Jun 25, 2026
29 checks passed
@wscurran wscurran added the NV QA Bugs found by the NVIDIA QA Team label Jun 26, 2026
@wscurran wscurran added UAT Issues flagged for User Acceptance Testing. VDR Linked to VDR finding labels Jun 26, 2026
cv pushed a commit that referenced this pull request Jun 26, 2026
…#5512) (#5541)

## Summary

Follow-up to #5537 addressing the orphan-backup symptom in #5512. When
the Docker GPU patch's recreate `docker run` fails **after** the
original sandbox was already renamed to
`*-nemoclaw-gpu-backup-<timestamp>`, the early-failure path removed only
the failed *new* container and left the backup orphaned — stranding the
sandbox with no live original, and colliding with
`*-nemoclaw-gpu-backup-*` on the next retry (as reported in #5512).

The supervisor-reconnect failure path already rolls back to the backup;
this path didn't.

## Fix

Reuse the existing rollback primitive (`rollbackToBackupContainer`) on
the early-failure path: remove the failed new container, rename the
backup back to the original name, and start it — restoring the pre-patch
sandbox instead of leaking a backup container.

- Adds `rollbackDockerGpuPatchOnRecreateFailure(refs, deps)` to
`docker-gpu-patch-finalize.ts`, which resolves the real `docker start` /
`docker rename` defaults (the recreate call path only carries a deps
subset, so `dockerStart` would otherwise be unset).
- Records `context.rolledBack` for failure diagnostics, matching the
reconnect-failure path.
- No `onboard.ts` change; all edits are under `src/lib/onboard/`.

## Testing

- New composed test in `docker-gpu-patch-rollback.test.ts`: when
`dockerRunDetached` fails, the backup is renamed back to the original
and started, and is never left as an orphaned container.
- `tsc -p tsconfig.src.json` clean; rollback / finalize / sandbox-create
suites pass (18/18).

## Relationship to the WSL Docker Desktop chain

- #5534 — gateway bind at `[2/8]`
- #5536 — gateway cleanup on probe failure
- #5537 — skip CDI GPU mode at `[6/8]` (makes the patch succeed on
Docker Desktop WSL, so this early-failure path is no longer hit there)
- this PR — restore the pre-patch sandbox for any *other* early
GPU-recreate failure

Refs #5512.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved recovery for Docker GPU patch recreation failures: if the
GPU-enabled recreate step fails after the original container is renamed,
the system now performs a reliable rollback to the pre-patch sandbox
state and cleans up the failed recreate attempt.

* **Tests**
* Added a rollback-path test for a recreate-phase `docker run --detach`
failure, verifying restoration of the original container name, restart
behavior, and correct cleanup (including ignoring the failed recreated
container).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>

---------

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Signed-off-by: Preksha Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>
@cv cv mentioned this pull request Jul 7, 2026
14 tasks
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…VIDIA#5537)

## Summary

On Docker Desktop + WSL2 with an NVIDIA GPU, onboard's `[6/8]` Docker
GPU patch recreates the sandbox container with `--device
nvidia.com/gpu=all` (CDI syntax) and fails:

```
CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all
```

even though preflight already logs that it will use the `--gpus`
compatibility path. The only workaround today is `--no-gpu` /
`NEMOCLAW_SANDBOX_GPU=0`, which disables GPU entirely.

## Root cause

Docker Desktop advertises CDI spec **directories**, so
`dockerReportsNvidiaCdiDevices()` returns true and
`buildDockerGpuModeCandidates()` offers CDI as the first candidate. The
create-only probe (`docker create … true`) passes, but the real recreate
fails because the WSL distro exposes **no usable `nvidia.com/gpu`
spec**. The Docker Desktop WSL status was detected at preflight but
never reached the mode selector — `selectDockerGpuPatchMode` only
received `{image, device, backend}`.

PR NVIDIA#5198 (which closed NVIDIA#5180) added the CDI-injection failure
classification, the `--no-gpu` recovery hint, and the warning that
`NEMOCLAW_DOCKER_GPU_PATCH=0` is ignored on this runtime — but it did
not change mode selection. This is the unaddressed root cause.

## Fix

Thread the existing Docker Desktop WSL detection
(`isDockerDesktopWslRuntime()`, already used to gate the patch) through
`selectDockerGpuPatchMode` into `buildDockerGpuModeCandidates`, and skip
the CDI candidate when on Docker Desktop WSL so the patch uses `--gpus
all` — the path preflight already commits to.

- Native Docker-CDI hosts are **unaffected**: they still prefer CDI,
preserving the gateway supervisor-wiring contract from NVIDIA#4948.
- The flag is resolved via the cached detector in
`docker-gpu-sandbox-create.ts`, so no change to `onboard.ts` and no
extra `docker info` calls.

## Testing

- New unit tests in `docker-gpu-patch-wsl.test.ts`: CDI is skipped
(first candidate is `--gpus all`) when `dockerDesktopWsl` is true even
with CDI advertised, and CDI is still preferred otherwise.
- `tsc -p tsconfig.src.json` clean; GPU-patch suites pass (remaining
failures are pre-existing Windows-only `/etc/cdi` path tests, identical
on `main`).

## Notes / follow-up

- This step is only reached after the `[2/8]` gateway-bind issue (NVIDIA#5513,
fix in NVIDIA#5534).
- Separate latent bug still open: on an **early** patch failure the
original sandbox is already renamed to
`*-nemoclaw-gpu-backup-<timestamp>` before container creation, and only
the new container is removed — leaving an orphan backup. Happy to follow
up with a focused PR for that cleanup.

Fixes NVIDIA#5512.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Enhanced GPU configuration for Docker Desktop on Windows Subsystem for
Linux (WSL). The system now properly detects WSL runtime environments
and automatically selects GPU acceleration modes that work reliably on
Docker Desktop WSL, avoiding GPU modes that may not be available or
incompatible within that specific environment.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…NVIDIA#5512) (NVIDIA#5541)

## Summary

Follow-up to NVIDIA#5537 addressing the orphan-backup symptom in NVIDIA#5512. When
the Docker GPU patch's recreate `docker run` fails **after** the
original sandbox was already renamed to
`*-nemoclaw-gpu-backup-<timestamp>`, the early-failure path removed only
the failed *new* container and left the backup orphaned — stranding the
sandbox with no live original, and colliding with
`*-nemoclaw-gpu-backup-*` on the next retry (as reported in NVIDIA#5512).

The supervisor-reconnect failure path already rolls back to the backup;
this path didn't.

## Fix

Reuse the existing rollback primitive (`rollbackToBackupContainer`) on
the early-failure path: remove the failed new container, rename the
backup back to the original name, and start it — restoring the pre-patch
sandbox instead of leaking a backup container.

- Adds `rollbackDockerGpuPatchOnRecreateFailure(refs, deps)` to
`docker-gpu-patch-finalize.ts`, which resolves the real `docker start` /
`docker rename` defaults (the recreate call path only carries a deps
subset, so `dockerStart` would otherwise be unset).
- Records `context.rolledBack` for failure diagnostics, matching the
reconnect-failure path.
- No `onboard.ts` change; all edits are under `src/lib/onboard/`.

## Testing

- New composed test in `docker-gpu-patch-rollback.test.ts`: when
`dockerRunDetached` fails, the backup is renamed back to the original
and started, and is never left as an orphaned container.
- `tsc -p tsconfig.src.json` clean; rollback / finalize / sandbox-create
suites pass (18/18).

## Relationship to the WSL Docker Desktop chain

- NVIDIA#5534 — gateway bind at `[2/8]`
- NVIDIA#5536 — gateway cleanup on probe failure
- NVIDIA#5537 — skip CDI GPU mode at `[6/8]` (makes the patch succeed on
Docker Desktop WSL, so this early-failure path is no longer hit there)
- this PR — restore the pre-patch sandbox for any *other* early
GPU-recreate failure

Refs NVIDIA#5512.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved recovery for Docker GPU patch recreation failures: if the
GPU-enabled recreate step fails after the original container is renamed,
the system now performs a reliable rollback to the pre-patch sandbox
state and cleans up the failed recreate attempt.

* **Tests**
* Added a rollback-path test for a recreate-phase `docker run --detach`
failure, verifying restoration of the original container name, restart
behavior, and correct cleanup (including ignoring the failed recreated
container).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>

---------

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Signed-off-by: Preksha Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression NV QA Bugs found by the NVIDIA QA Team platform: wsl Affects Windows Subsystem for Linux UAT Issues flagged for User Acceptance Testing. VDR Linked to VDR finding

Projects

None yet

5 participants