Skip to content
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
45fdef7
feat(gpu): prefer native OpenShell with compatibility fallback
ericksoa Jul 6, 2026
f2f5e6f
fix(gpu): harden native fallback orchestration
ericksoa Jul 6, 2026
4a7f219
fix(gpu): narrow and modularize fallback orchestration
ericksoa Jul 6, 2026
3f11600
test(gpu): keep recreate DNS fixtures linear
ericksoa Jul 6, 2026
10e615b
refactor(gpu): split fallback review boundaries
ericksoa Jul 6, 2026
4829c80
docs(gpu): clarify DNS fallback boundary
ericksoa Jul 6, 2026
8d6edaa
test(gpu): make fallback boundaries explicit
ericksoa Jul 6, 2026
435a451
test(gpu): exercise default diagnostics capture
ericksoa Jul 6, 2026
e9dd4ae
refactor(gpu): tighten fallback review boundaries
ericksoa Jul 6, 2026
a5ef521
fix(gpu): wait for stable fallback readiness
ericksoa Jul 6, 2026
d01abe1
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 6, 2026
24c1835
refactor(gpu): isolate fallback trust boundaries
ericksoa Jul 6, 2026
5eed49d
refactor(gpu): make review contracts explicit
ericksoa Jul 6, 2026
881a781
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 6, 2026
2bb45f6
test(gpu): clarify fallback safety contracts
ericksoa Jul 6, 2026
f214cbf
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 6, 2026
4d3ca8e
fix(gpu): redact cleanup refusal diagnostics
ericksoa Jul 6, 2026
1fe1181
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 7, 2026
e599ade
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 7, 2026
176201d
fix(gpu): harden native fallback trust boundary
ericksoa Jul 7, 2026
abc26da
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 7, 2026
ebc37dc
test(gpu): satisfy source-shape guard
ericksoa Jul 7, 2026
6c6b8e1
test(gpu): isolate fallback runtime fixture
ericksoa Jul 8, 2026
8793c25
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 8, 2026
1e443c3
test(gpu): cover rebuild route selection
ericksoa Jul 8, 2026
d58a99a
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 8, 2026
1434cb4
ci(e2e): trap GPU fixture Docker restore
ericksoa Jul 8, 2026
3e82fd2
Merge remote-tracking branch 'origin/main' into feat/gpu-native-fallb…
ericksoa Jul 8, 2026
cf3eac4
merge(main): refresh GPU fallback branch
cv Jul 10, 2026
fd65f6f
test(e2e): map Hermes GPU mock parity
cv Jul 10, 2026
83ceaba
test(onboard): cover fallback rebuild preflight
cv Jul 10, 2026
16a283c
docs(gpu): clarify fallback controls
cv Jul 10, 2026
200d211
docs(gpu): clarify rebuild route resolution
cv Jul 10, 2026
461ac0a
merge(main): refresh PR branch
cv Jul 11, 2026
ce6c748
Merge branch 'main' into feat/gpu-native-fallback-6110
cv Jul 11, 2026
7674c5b
merge(main): refresh PR branch
cv Jul 11, 2026
da57b58
Merge branch 'main' into feat/gpu-native-fallback-6110
cv Jul 11, 2026
5efbbcc
Merge branch 'main' into feat/gpu-native-fallback-6110
cv Jul 11, 2026
0bb94fe
Merge branch 'main' into feat/gpu-native-fallback-6110
cv Jul 11, 2026
2521525
Merge remote-tracking branch 'origin/main' into codex/pr-6333-green
ericksoa Jul 12, 2026
6891a65
fix(gpu): harden fallback review boundaries
ericksoa Jul 12, 2026
cf10c5c
test(gpu): keep route cases linear
ericksoa Jul 12, 2026
fa88477
fix(e2e): trust privileged GPU fixture
ericksoa Jul 12, 2026
cb31b8b
refactor(e2e): compact Hermes GPU boundary checks
ericksoa Jul 12, 2026
6252c55
fix(e2e): harden GPU fallback trust boundaries
ericksoa Jul 12, 2026
c75949f
refactor(e2e): compact GPU boundary coverage
ericksoa Jul 12, 2026
b38165c
test(e2e): keep GPU trust checks focused
ericksoa Jul 12, 2026
675f3ea
fix(onboard): defer compatibility network mutation
ericksoa Jul 12, 2026
26d8a22
Merge remote-tracking branch 'origin/main' into codex/pr-6333-green
ericksoa Jul 12, 2026
de7b367
test(onboard): consolidate GPU fallback coverage
ericksoa Jul 12, 2026
324516e
test(onboard): keep diagnostics cases branchless
ericksoa Jul 12, 2026
0abe4c9
test(onboard): trim GPU review context
ericksoa Jul 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 132 additions & 2 deletions .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1841,19 +1841,25 @@ jobs:
if: ${{ contains(format(',{0},', inputs.jobs), ',hermes-gpu-startup,') || contains(format(',{0},', inputs.targets), ',hermes-gpu-startup,') }}
runs-on: linux-amd64-gpu-rtxpro6000-latest-1
timeout-minutes: 75
strategy:
fail-fast: false
max-parallel: 1
matrix:
scenario: [native, fallback, compatibility-only]
env:
E2E_JOB: "1"
E2E_DEFAULT_ENABLED: "0"
E2E_TARGET_ID: "hermes-gpu-startup"
E2E_ARTIFACT_DIR: ${{ github.workspace }}/e2e-artifacts/live/hermes-gpu-startup
E2E_ARTIFACT_DIR: ${{ github.workspace }}/e2e-artifacts/live/hermes-gpu-startup/${{ matrix.scenario }}
E2E_HERMES_GPU_STARTUP_SCENARIO: ${{ matrix.scenario }}
NEMOCLAW_CLI_BIN: ${{ github.workspace }}/bin/nemoclaw.js
NEMOCLAW_RUN_LIVE_E2E: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_AGENT: hermes
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_RECREATE_SANDBOX: "1"
NEMOCLAW_SANDBOX_GPU: "1"
NEMOCLAW_SANDBOX_NAME: e2e-hermes-gpu-startup
NEMOCLAW_SANDBOX_NAME: e2e-hermes-gpu-startup-${{ matrix.scenario }}
NEMOCLAW_ONBOARD_VALIDATION_TIMEOUT_SECONDS: "60"
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -1867,15 +1873,139 @@ jobs:
uses: NVIDIA/NemoClaw/.github/actions/prepare-e2e@50281ee84c4a6fc759da95ea28fc0b7d9c378a28

- name: Run Hermes GPU startup live Vitest test
shell: bash
run: |
set -euo pipefail

wait_for_docker() {
local failure_message="$1"
for attempt in $(seq 1 30); do
if docker info >/dev/null 2>&1; then
return 0
fi
if [ "$attempt" -eq 30 ]; then
echo "$failure_message" >&2
return 1
fi
sleep 2
done
}

if [ "$E2E_HERMES_GPU_STARTUP_SCENARIO" = fallback ]; then
state_dir="$RUNNER_TEMP/hermes-gpu-fallback-docker-runtime"
daemon_json=/etc/docker/daemon.json
rm -rf "$state_dir"
mkdir -p "$state_dir" "$E2E_ARTIFACT_DIR"
sudo -n true

# invalidState: a cancelled fallback test leaves the runner-global Docker default
# runtime changed, contaminating later OpenShell jobs on this self-hosted runner.
# sourceBoundary: this test-only step owns the daemon mutation, the Vitest process,
# and exact restoration of the pre-test daemon config and effective runtime.
# whyNotSourceFix: production must fail closed when GPU attachment is ambiguous;
# this fixture needs a real no-GPU partial container on an nvidia-default runner.
# regressionTest: hermes-gpu-startup-workflow-boundary requires this same-step trap.
# removalCondition: remove the daemon fixture when OpenShell can create a provably
# GPU-unattached partial sandbox without consulting Docker's default runtime.
restore_docker_default_runtime() {
local command_status=$?
local restore_failed=0
local original_runtime=""
local restored_runtime=""
trap - EXIT INT TERM
set +e

if [ -f "$state_dir/default-runtime.modified" ]; then
if [ -f "$state_dir/daemon.json.absent" ]; then
sudo rm -f "$daemon_json" || restore_failed=1
else
sudo install -m 0644 "$state_dir/daemon.json.original" "$daemon_json" || restore_failed=1
fi
if [ "$restore_failed" -eq 0 ]; then
sudo systemctl restart docker || restore_failed=1
fi
if [ "$restore_failed" -eq 0 ]; then
wait_for_docker "Docker did not recover while restoring its original default runtime" || restore_failed=1
fi
fi

if [ "$restore_failed" -eq 0 ]; then
original_runtime="$(cat "$state_dir/default-runtime.original")" || restore_failed=1
fi
if [ "$restore_failed" -eq 0 ]; then
restored_runtime="$(docker info --format '{{.DefaultRuntime}}')" || restore_failed=1
fi
if [ "$restore_failed" -eq 0 ] && [ "$restored_runtime" != "$original_runtime" ]; then
echo "Docker default runtime was not restored: expected $original_runtime, got $restored_runtime" >&2
restore_failed=1
fi
if [ "$restore_failed" -eq 0 ] && [ -f "$state_dir/daemon.json.absent" ]; then
sudo test ! -e "$daemon_json" || restore_failed=1
elif [ "$restore_failed" -eq 0 ]; then
sudo cmp -s "$state_dir/daemon.json.original" "$daemon_json" || restore_failed=1
fi
if [ -n "$restored_runtime" ]; then
printf '%s\n' "$restored_runtime" >"$E2E_ARTIFACT_DIR/docker-default-runtime-restored.txt" || restore_failed=1
fi
if [ "$restore_failed" -eq 0 ]; then
rm -rf "$state_dir" || restore_failed=1
fi
if [ "$restore_failed" -ne 0 ]; then
echo "Failed to restore the Docker daemon after the fallback fixture" >&2
exit 1
fi
exit "$command_status"
}

# OpenShell leaves Runtime unset after the wrapper strips --gpu, so Docker
# otherwise substitutes its daemon default. Force runc only within this
# trap-guarded process so production can prove the partial container has no GPU.
original_runtime="$(docker info --format '{{.DefaultRuntime}}')"
printf '%s\n' "$original_runtime" >"$state_dir/default-runtime.original"
printf '%s\n' "$original_runtime" >"$E2E_ARTIFACT_DIR/docker-default-runtime-before.txt"
if sudo test -f "$daemon_json"; then
sudo cat "$daemon_json" | tee "$state_dir/daemon.json.original" >/dev/null
else
: >"$state_dir/daemon.json.absent"
printf '{}\n' >"$state_dir/daemon.json.original"
fi
trap restore_docker_default_runtime EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

if [ "$original_runtime" != runc ]; then
node - "$state_dir/daemon.json.original" "$state_dir/daemon.json.runc" <<'NODE'
const fs = require("node:fs");
const [inputPath, outputPath] = process.argv.slice(2);
const source = fs.readFileSync(inputPath, "utf8").trim();
const config = source ? JSON.parse(source) : {};
if (config === null || Array.isArray(config) || typeof config !== "object") {
throw new Error("Docker daemon.json must contain a top-level object");
}
config["default-runtime"] = "runc";
fs.writeFileSync(outputPath, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 });
NODE

: >"$state_dir/default-runtime.modified"
sudo install -m 0644 "$state_dir/daemon.json.runc" "$daemon_json"
sudo systemctl restart docker
wait_for_docker "Docker did not recover after selecting the runc default runtime"
fi
test "$(docker info --format '{{.DefaultRuntime}}')" = runc
docker info --format '{{json .Runtimes}}' | grep -q 'nvidia'
printf '%s\n' runc >"$E2E_ARTIFACT_DIR/docker-default-runtime-during.txt"
fi

npx vitest run --project e2e-live \
test/e2e/live/hermes-gpu-startup.test.ts \
--silent=false --reporter=default --reporter=test/e2e/risk-signal-reporter.ts

- name: Upload Hermes GPU startup artifacts
if: always()
uses: NVIDIA/NemoClaw/.github/actions/upload-e2e-artifacts@7768e15eb90d3ee2d33432f481dfe8747e4f6d57
with:
name: e2e-hermes-gpu-startup-${{ matrix.scenario }}
path: e2e-artifacts/live/hermes-gpu-startup/${{ matrix.scenario }}/

- name: Clean up Docker auth
if: always()
Expand Down
26 changes: 17 additions & 9 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -719,12 +719,18 @@ Use `--no-gpu` to opt out when you want host-side inference providers only and d
Use `--gpu` to require GPU passthrough and fail fast if an NVIDIA GPU is not detected.
Use `--sandbox-gpu` or `--no-sandbox-gpu` to control only direct NVIDIA GPU access inside the sandbox.
Use `--sandbox-gpu --sandbox-gpu-device <device>` to pass a specific OpenShell GPU device selector to `openshell sandbox create`; device selectors require explicit sandbox GPU enablement.
On ordinary native Linux Docker-driver hosts with usable CDI, NemoClaw uses OpenShell native GPU injection by default.
On Docker Desktop WSL and Jetson/Tegra, NemoClaw creates the sandbox first and then recreates the OpenShell-managed Docker container with NVIDIA GPU access by default.
When you force this compatibility path on ordinary native Linux, NemoClaw uses an available NVIDIA CDI spec before falling back to Docker `--gpus all` or the NVIDIA runtime.
On ordinary native Linux Docker-driver hosts, NemoClaw uses native OpenShell GPU injection by default and never broadens confinement automatically.
Set `NEMOCLAW_DOCKER_GPU_PATCH=fallback` to explicitly authorize one native attempt followed by one compatibility retry.
NemoClaw permits the retry only after it confirms either a trusted host-side GPU routing failure or an explicit driver proof plus exact-container host configuration showing that no GPU was attached.
It then saves redacted diagnostics and removes the incomplete sandbox before retrying.
Sandbox-reported CUDA output alone never authorizes the broader compatibility envelope, even when the operator enabled fallback.
That case fails closed and points to the explicit `NEMOCLAW_DOCKER_GPU_PATCH=1` compatibility-only control.
NemoClaw retries only after it verifies that no OpenShell-managed Docker container labeled for that sandbox remains; if cleanup cannot be proven safe, onboarding stops and prints cleanup guidance instead.
On Docker Desktop WSL and Jetson/Tegra, automatic GPU onboarding uses the compatibility path directly.
On ordinary native Linux, the compatibility path uses an available NVIDIA CDI spec before falling back to Docker `--gpus all` or the NVIDIA runtime.
On Docker Desktop WSL, the compatibility path skips CDI and tries Docker `--gpus all` before the NVIDIA runtime.
On Jetson/Tegra hosts, the compatibility path uses the NVIDIA runtime and adds the host group IDs that own `/dev/nvmap` and `/dev/nvhost-*` so the sandbox user can initialize CUDA.
If the patch fails, onboarding keeps diagnostics and prints a manual cleanup command rather than deleting the failed sandbox automatically.
If the compatibility attempt fails, onboarding keeps its diagnostics and the failed sandbox in place and prints a manual cleanup command.

Prerequisites:

Expand All @@ -737,9 +743,10 @@ When GPU passthrough is enabled and a gateway already exists without it, onboard
If no other registered sandbox depends on that gateway, or if `--recreate-sandbox` is recreating the only registered sandbox with the same name, onboarding cleans up the stale gateway and continues.
If other sandboxes depend on the gateway or Docker state is unclear, onboarding exits without cleanup and prints targeted destroy or gateway-removal guidance.
To add GPU to an existing sandbox, rerun with `--recreate-sandbox`.
Leave `NEMOCLAW_DOCKER_GPU_PATCH` unset or set it to `auto` to use the platform default.
Set `NEMOCLAW_DOCKER_GPU_PATCH=1` to force the legacy Docker container-swap path on ordinary native Linux.
Set `NEMOCLAW_DOCKER_GPU_PATCH=0` to select native OpenShell GPU injection on ordinary native Linux or Jetson/Tegra.
Leave `NEMOCLAW_DOCKER_GPU_PATCH` unset or set it to `auto` for native-only GPU onboarding on ordinary native Linux.
Set `NEMOCLAW_DOCKER_GPU_PATCH=fallback` to explicitly opt into one bounded native-to-compatibility retry on ordinary native Linux.
Set `NEMOCLAW_DOCKER_GPU_PATCH=0` to require native OpenShell GPU injection on ordinary native Linux or Jetson/Tegra.
Set `NEMOCLAW_DOCKER_GPU_PATCH=1`, or another legacy nonzero value, to use only the compatibility path on ordinary native Linux.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Use `NEMOCLAW_DOCKER_GPU_PATCH=0` on Jetson/Tegra only for troubleshooting because it bypasses Tegra device-group propagation and CUDA may not initialize.
Docker Desktop WSL ignores `NEMOCLAW_DOCKER_GPU_PATCH=0` because GPU passthrough on that runtime requires the compatibility patch.
Use `--no-sandbox-gpu`, `--no-gpu`, or `NEMOCLAW_SANDBOX_GPU=0` when you want to disable sandbox GPU passthrough on Docker Desktop WSL.
Expand Down Expand Up @@ -2370,7 +2377,8 @@ The command backs up workspace state, destroys the old sandbox (including its ho
Credentials are stripped from backups before storage.
Policy presets applied to the old sandbox are reapplied to the new one so your egress rules survive the rebuild.
The replacement uses the recorded compatible-endpoint reasoning mode and web search selection instead of ambient shell values.
The recorded sandbox GPU mode is preserved across rebuild.
Rebuild preserves the recorded sandbox GPU enablement mode and, for an explicitly enabled sandbox, its recorded device selector.
It re-resolves the Docker-driver GPU route from the current host and current `NEMOCLAW_DOCKER_GPU_PATCH` value, so native-only, explicitly authorized native-with-fallback, and compatibility-only routing may differ from the original onboarding run.
A rebuild preserves the recorded tool-disclosure mode unless `--tool-disclosure` explicitly changes it; it ignores an ambient `NEMOCLAW_TOOL_DISCLOSURE` value while recreating the sandbox.
A rebuild preserves the recorded Deep Agents Code observability choice and matching local OTLP policy state unless `--observability` or `--no-observability` explicitly changes them.
A rebuild preserves the recorded Deep Agents Code auto-approval capability unless `--dcode-auto-approval` explicitly changes it.
Expand Down Expand Up @@ -3363,7 +3371,7 @@ Set them before running `$$nemoclaw onboard`.
| `NEMOCLAW_SANDBOX_GPU_DEVICE` | OpenShell GPU device selector | Selects the GPU device passed with `openshell sandbox create --gpu-device`. Requires explicit sandbox GPU enablement with `NEMOCLAW_SANDBOX_GPU=1` (or `--sandbox-gpu` for CLI-driven onboarding); otherwise onboarding rejects the selector instead of treating it as an implicit opt-in. |
| `NEMOCLAW_SANDBOX_BASE_IMAGE_REFRESH` | `1`, `true`, `yes`, or `on` to enable | Bypasses recorded sandbox base-image resolution metadata during onboarding, recreation, and rebuild. NemoClaw reruns candidate resolution but can still use a compatible image from Docker's local image store. Versioned release candidates that exist locally but fail validation are refreshed from the registry once during normal resolution. This setting does not discard onboarding session state. |
| `NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD` | unset or `auto` (default); `1`, `true`, `yes`, or `on` to enable; `0`, `false`, `no`, or `off` to disable | Controls whether base-image resolution may build a compatible image locally. The default allows builds during normal CLI runs and disables them when `NODE_ENV=test` or `VITEST=true`. When source inputs or a missing/incompatible release-version base require a fresh build, disabling local builds makes resolution fail instead of using an unproven image. |
| `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `1`, or `0` | Selects Linux Docker-driver GPU routing. Unset or `auto` uses native OpenShell GPU injection on ordinary native Linux and the compatibility patch on Docker Desktop WSL and Jetson/Tegra. `1` forces the compatibility patch. `0` selects native injection on ordinary native Linux and Jetson/Tegra, but Docker Desktop WSL ignores it. On Jetson/Tegra, use `0` only for troubleshooting because it bypasses the device-group propagation needed for CUDA. |
| `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `fallback`, `1`, or `0`; legacy nonzero values remain temporarily accepted for compatibility | Selects Linux Docker-driver GPU routing. Unset, `auto`, or `0` uses native OpenShell GPU injection on ordinary native Linux. `fallback` explicitly opts into one native attempt followed by one bounded compatibility retry when trusted host evidence identifies a GPU-routing failure. `1` and legacy nonzero values select the compatibility patch from the outset. Docker Desktop WSL and Jetson/Tegra use the compatibility path by default; Docker Desktop WSL ignores `0`, while Jetson/Tegra accepts `0` only as a troubleshooting override that bypasses device-group propagation. |
| `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH` | `1` to enable; disabled by default | This setting explicitly opts into the Linux gateway compatibility container for an older host ABI or a diagnostic run; use it only on a trusted local host because it uses host networking and mounts the Docker socket read-only even though the socket still exposes the privileged Docker API; prefer OpenShell 0.0.72's directly supported glibc 2.28+ path; see the [OpenShell 0.0.72 compatibility review](/user-guide/openclaw/security/openshell-0.0.72-compatibility-review#source-of-truth-boundaries) for details. |
| `NEMOCLAW_OPENSHELL_GATEWAY_BIN` | path | Advanced override for the `openshell-gateway` binary used by the Linux Docker-driver standalone fallback. Defaults to the binary next to `openshell`, then common install paths. |
| `NEMOCLAW_OPENSHELL_SANDBOX_BIN` | path | Advanced override for the `openshell-sandbox` binary used by the Linux Docker-driver standalone fallback. Defaults to the binary next to `openshell`, then common install paths. |
Expand Down
Loading
Loading