Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
efb5fa7
fix(onboard): grant Jetson nvmap group write access
sandl99 Aug 3, 2026
09bef64
docs(troubleshooting): simplify Jetson nvmap guidance
sandl99 Aug 3, 2026
632a5c9
test(onboard): cover failed nvmap permission verification
sandl99 Aug 3, 2026
0bb4a74
docs(troubleshooting): clarify nvmap rule lifecycle
sandl99 Aug 3, 2026
ea03eab
test(onboard): cover missing Jetson nvmap device
sandl99 Aug 3, 2026
c378434
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 4, 2026
29262e9
fix(onboard): preserve Jetson sandbox GPU access
sandl99 Aug 4, 2026
2257f36
Merge remote-tracking branch 'origin/main' into fix/jetson-nvmap-nonroot
sandl99 Aug 4, 2026
c9a80f8
fix(onboard): scope Jetson nvmap access to OpenClaw
sandl99 Aug 4, 2026
457d1f9
refactor(onboard): keep entrypoint line budget neutral
sandl99 Aug 4, 2026
9bd5a3a
test(images): stage Jetson bootstrap helper
sandl99 Aug 4, 2026
67a8579
fix(images): stage Jetson bootstrap script
sandl99 Aug 4, 2026
3690a12
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 4, 2026
ed7f1ec
fix(onboard): preserve managed Jetson groups
sandl99 Aug 4, 2026
1a17a1c
Merge branch 'main' into fix/jetson-nvmap-nonroot
apurvvkumaria Aug 4, 2026
8785891
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 5, 2026
5237552
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 5, 2026
92e10e2
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 5, 2026
5a92cb1
Merge remote-tracking branch 'origin/main' into fix/jetson-nvmap-nonroot
sandl99 Aug 5, 2026
70600b3
fix(installer): configure Jetson nvmap before L4T classification
sandl99 Aug 5, 2026
25d2c8b
fix(onboard): verify Jetson nvmap access before sandbox creation
sandl99 Aug 5, 2026
e9198bc
test(jetson): add nvmap boundary proof
sandl99 Aug 5, 2026
c7d6e05
fix(test): initialize Jetson POC readonly values
sandl99 Aug 5, 2026
f27f724
fix(test): make Jetson nvmap proof fail closed
apurvvkumaria Aug 5, 2026
8508897
Merge remote-tracking branch 'origin/main' into fix/jetson-nvmap-nonroot
sandl99 Aug 5, 2026
bf27905
fix(onboard): grant Jetson devices in sandbox policy
sandl99 Aug 5, 2026
98e392e
test(jetson): add CUDA boundary probe
sandl99 Aug 5, 2026
ba6080d
merge: resolve conflicts with main
github-actions[bot] Aug 6, 2026
06d7407
chore(onboard): revert prior Jetson GPU changes
sandl99 Aug 6, 2026
da3d48e
merge: sync main before focused Jetson fix
sandl99 Aug 6, 2026
000e88e
fix(onboard): permit Jetson GPU policy paths
sandl99 Aug 6, 2026
48338b7
fix(onboard): apply Jetson policy to default OpenClaw
sandl99 Aug 6, 2026
deee7ef
fix(onboard): preserve Jetson device groups
sandl99 Aug 6, 2026
a6bf7b5
fix(onboard): stage Jetson group bootstrap
sandl99 Aug 6, 2026
8255118
chore(onboard): add Jetson GPU diagnostic
sandl99 Aug 6, 2026
7fad34a
chore(onboard): add Jetson device-group proof
sandl99 Aug 6, 2026
0a65abe
fix(onboard): preserve Jetson groups in compatibility path
sandl99 Aug 6, 2026
4f65114
Merge remote-tracking branch 'origin/main' into fix/7610-pr-revert
sandl99 Aug 6, 2026
d6cfa87
refactor(onboard): carry agent scope with patch options
sandl99 Aug 6, 2026
03529c5
fix(images): audit Jetson wrapper metadata check
sandl99 Aug 6, 2026
7757675
chore(onboard): add cuInit boundary diagnostic
sandl99 Aug 6, 2026
f0a6720
chore(onboard): prove OpenRM policy boundary
sandl99 Aug 6, 2026
1964870
fix(onboard): make OpenRM proof non-interactive
sandl99 Aug 6, 2026
2b4a89a
fix(onboard): run OpenRM proof before rollback
sandl99 Aug 6, 2026
f06f365
fix(onboard): resolve Jetson review findings
apurvvkumaria Aug 6, 2026
b319e3f
Merge remote-tracking branch 'origin/main' into codex/pr8099-feedback…
apurvvkumaria Aug 6, 2026
cd42a42
test(images): cover Jetson helper permissions
apurvvkumaria Aug 6, 2026
0a0bc13
fix(onboard): recreate sandbox for OpenRM proof
sandl99 Aug 6, 2026
6e1cb76
fix(onboard): honor explicit sandbox recreation
sandl99 Aug 6, 2026
5a0fa51
chore(onboard): expand Jetson OpenRM proof
sandl99 Aug 6, 2026
d9cbe35
docs(onboard): describe OpenRM proof matrix
sandl99 Aug 6, 2026
f25c2b5
fix(onboard): validate Jetson groups before mutation
apurvvkumaria Aug 6, 2026
7b70ce2
fix(onboard): fail closed on OpenRM restore
apurvvkumaria Aug 6, 2026
e0e1e9c
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 7, 2026
8fde81f
test(onboard): make Jetson policy proof standalone
sandl99 Aug 7, 2026
72dd2f3
test(onboard): enable standalone OpenRM matrix
sandl99 Aug 7, 2026
21f21ee
test(onboard): cover invalid group database
apurvvkumaria Aug 7, 2026
ceff582
test(onboard): isolate Jetson process confinement
sandl99 Aug 7, 2026
3c81134
test(onboard): isolate remaining Jetson boundary
sandl99 Aug 7, 2026
60339b6
fix(onboard): allow Jetson NvSci IPC device
sandl99 Aug 7, 2026
250b7b4
fix(onboard): remove unproven NvSci policy
sandl99 Aug 7, 2026
f26b486
merge: resolve conflicts with main
github-actions[bot] Aug 8, 2026
65adec8
Merge branch 'main' into fix/jetson-nvmap-nonroot
apurvvkumaria Aug 10, 2026
a8ca0a0
Merge branch 'main' into fix/jetson-nvmap-nonroot
sandl99 Aug 11, 2026
1c723ba
test(onboard): expand Jetson CUDA diagnostics
sandl99 Aug 11, 2026
5e2459a
Revert "test(onboard): expand Jetson CUDA diagnostics"
sandl99 Aug 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,12 @@ The installer auto-detects NVIDIA Jetson devices (Orin and Thor) and applies req
If the Jetson setup step fails, verify that you have `sudo` access and that Docker is installed and running.

For JetPack 6 (L4T 36.x), the setup switches iptables to legacy mode and adjusts the Docker daemon configuration.
The setup also grants the existing `/dev/nvmap` owning group read-write access.
It writes the NemoClaw-managed `/etc/udev/rules.d/99-zz-nemoclaw-nvmap.rules` rule with mode `0660`, reloads the udev rules, and updates the live device.
The rule applies the mode when udev recreates `/dev/nvmap`, including after a host reboot.
This change grants write access to every member of the existing device-owning group, but it does not grant access to other users or run the sandbox as root.
If `/dev/nvmap` is absent, the setup warns, skips the rule, and continues installation.
CUDA can still fail until the device exists and you rerun the installer.
For JetPack 7 (L4T 38.x / Thor), only bridge netfilter and sysctl settings are applied.
For JetPack 7 (L4T 39.x), bridge netfilter is loaded only when the host is missing it.
Some R39 images already ship with `br_netfilter` configured and are left untouched.
Expand Down Expand Up @@ -2510,8 +2516,64 @@ To skip GPU passthrough entirely, rerun with `--no-gpu` or set `NEMOCLAW_SANDBOX

Automatic GPU onboarding uses the compatibility path directly; it does not make a native attempt first.
The path recreates the OpenShell-managed Docker container with NVIDIA GPU flags and propagates eligible host group IDs for the supported Jetson GPU device nodes.
NemoClaw propagates a device group only when the host node grants that group both read and write access and does not grant both permissions to other users.
Use `NEMOCLAW_DOCKER_GPU_PATCH=0` only for troubleshooting because it bypasses that group propagation and CUDA may not initialize.

If CUDA reports `NvRmMemInitNvmap failed with Permission denied` and `cuInit(0)=999`, inspect the host device:

```bash
stat -c '%A %U:%G %n' /dev/nvmap
```

The group permission characters must be `rw`, such as the group field in `crw-rw----`.
On JetPack 6, rerun the NemoClaw installer to apply the persistent udev rule and update the live device:

<Warning>
The installer grants write access to every member of the existing `/dev/nvmap` owning group.
It also persists mode `0660` when udev recreates the device.
</Warning>

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

Verify the rule content and the live device permissions:

```bash
grep -Fx 'KERNEL=="nvmap", MODE="0660"' /etc/udev/rules.d/99-zz-nemoclaw-nvmap.rules
stat -c '%A %U:%G %n' /dev/nvmap
```

The first command must print the exact rule.
The group permission characters from the second command must be `rw`.

<Warning>
The following temporary remediation grants write access to every member of the existing `/dev/nvmap` owning group.
The permission lasts until udev recreates the device, such as during a host reboot.
</Warning>

If you need to diagnose the failure before rerunning the installer, update the live device and verify its group permissions:

```bash
sudo chmod g+rw /dev/nvmap
stat -c '%A %U:%G %n' /dev/nvmap
```

Clean up the failed sandbox with the command from the onboarding output, then rerun onboarding.

NemoClaw uninstall does not remove the Jetson host rule.
If you no longer need non-root Jetson GPU passthrough, remove the rule and reload udev:

```bash
sudo rm /etc/udev/rules.d/99-zz-nemoclaw-nvmap.rules
sudo udevadm control --reload-rules
test ! -e /etc/udev/rules.d/99-zz-nemoclaw-nvmap.rules && echo "NemoClaw nvmap rule removed"
```

The final command must print `NemoClaw nvmap rule removed`.
Removing the rule does not change the live device mode.
The remaining platform udev rules determine the mode after udev recreates `/dev/nvmap`, such as during a host reboot.

#### Common compatibility-path recovery

If the compatibility attempt fails on any host, onboarding leaves the failed sandbox and diagnostic bundle in place so you can inspect the OpenShell and Docker state.
Expand Down
31 changes: 31 additions & 0 deletions scripts/setup-jetson.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ set -euo pipefail
SUDO=()
((EUID != 0)) && SUDO=(sudo)

NVMAP_DEVICE="/dev/nvmap"
NVMAP_UDEV_RULE="/etc/udev/rules.d/99-zz-nemoclaw-nvmap.rules"
NVMAP_UDEV_RULE_CONTENT='KERNEL=="nvmap", MODE="0660"'

info() {
printf "[INFO] %s\n" "$*"
}
Expand Down Expand Up @@ -56,6 +60,32 @@ apply_br_netfilter_setup() {
echo "net.bridge.bridge-nf-call-iptables=1" | "${SUDO[@]}" tee /etc/sysctl.d/99-nemoclaw.conf >/dev/null
}

configure_nvmap_group_access() {
local device_state device_type verified_state verified_permissions

if ! device_state="$(LC_ALL=C stat -c '%F|%A' "$NVMAP_DEVICE" 2>/dev/null)"; then
warn "JetPack 6 host setup could not find $NVMAP_DEVICE. Non-root sandbox CUDA can fail until this device exists."
return 0
fi

device_type="${device_state%%|*}"
[[ "$device_type" == "character special file" ]] \
|| error "$NVMAP_DEVICE must be a character device before NemoClaw changes its group permissions."

warn "JetPack 6 host setup grants every member of the existing $NVMAP_DEVICE owning group write access and persists mode 0660 when udev recreates the device."
printf '%s\n' "$NVMAP_UDEV_RULE_CONTENT" | "${SUDO[@]}" tee "$NVMAP_UDEV_RULE" >/dev/null
"${SUDO[@]}" udevadm control --reload-rules
"${SUDO[@]}" chmod g+rw "$NVMAP_DEVICE"

verified_state="$(LC_ALL=C stat -c '%F|%A' "$NVMAP_DEVICE" 2>/dev/null)" \
|| error "Could not verify $NVMAP_DEVICE after granting group read-write access."
IFS='|' read -r device_type verified_permissions <<<"$verified_state"
[[ "$device_type" == "character special file" && "${verified_permissions:4:2}" == "rw" ]] \
|| error "$NVMAP_DEVICE does not grant its owning group read-write access after host setup."

info "$NVMAP_DEVICE grants its owning group read-write access. The udev rule $NVMAP_UDEV_RULE preserves this mode after reboot."
}

warn_host_setup_skipped() {
warn "Skipped Jetson host setup: iptables legacy mode and the Docker daemon.json adjustment (L4T 36.x only), and br_netfilter with net.bridge.bridge-nf-call-iptables=1 (every release)."
warn "Without br_netfilter, k3s inside the OpenShell gateway cannot NAT sandbox pod traffic to ClusterIP services, so sandbox pods cannot reach CoreDNS."
Expand Down Expand Up @@ -188,6 +218,7 @@ except Exception:
os.unlink(tmp)
raise
PYEOF
configure_nvmap_group_access
;;
jp7-r38)
# JP7 R38 does not need iptables or Docker daemon.json changes.
Expand Down
11 changes: 11 additions & 0 deletions src/lib/onboard/docker-gpu-jetson-groups.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ describe("detectTegraDeviceGroupGids", () => {
{ gid: 0, mode: 0o660 },
{ gid: 2_147_483_648, mode: 0o660 },
{ gid: 44, mode: 0o600 },
{ gid: 44, mode: 0o440 },
{ gid: 44, mode: 0o620 },
{ gid: 104, mode: 0o666 },
];
let index = 0;
Expand All @@ -47,6 +49,15 @@ describe("detectTegraDeviceGroupGids", () => {
).toEqual([]);
});

it("does not treat a read-only nvmap group as CUDA access (#7610)", () => {
expect(
detectTegraDeviceGroupGids({
statDeviceAccess: () => ({ gid: 44, mode: 0o440 }),
listDevicePaths: () => ["/dev/nvmap"],
}),
).toEqual([]);
});

it("returns no GIDs when Tegra nodes are missing or unreadable", () => {
expect(
detectTegraDeviceGroupGids({
Expand Down
6 changes: 4 additions & 2 deletions src/lib/onboard/docker-gpu-jetson-groups.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,15 @@ export function detectTegraDeviceGroupGids(
const gid = access?.gid ?? null;
const groupAccessBits = access === null ? 0 : (access.mode >> 3) & READ_WRITE_PERMISSION_BITS;
const otherAccessBits = access === null ? 0 : access.mode & READ_WRITE_PERMISSION_BITS;
const groupAddsAccess = (groupAccessBits & ~otherAccessBits) !== 0;
const groupAddsReadWriteAccess =
groupAccessBits === READ_WRITE_PERMISSION_BITS &&
otherAccessBits !== READ_WRITE_PERMISSION_BITS;
if (
gid !== null &&
Number.isSafeInteger(gid) &&
gid > 0 &&
gid <= MAX_DOCKER_SUPPLEMENTARY_GID &&
groupAddsAccess
groupAddsReadWriteAccess
) {
gids.add(String(gid));
}
Expand Down
2 changes: 1 addition & 1 deletion src/lib/onboard/docker-gpu-patch-recreate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ export function recreateOpenShellDockerSandboxContainer(
);
} else {
console.warn(
" ⚠ Could not resolve the group owning Jetson Tegra GPU device nodes (/dev/nvmap); CUDA may fail with NvRmMemInitNvmap permission denied. Confirm /dev/nvmap exists and is group-readable on the host.",
" ⚠ Could not resolve a read-write group for Jetson Tegra GPU device nodes (/dev/nvmap); CUDA may fail with NvRmMemInitNvmap permission denied. Confirm /dev/nvmap exists and grants its owning group read-write access on the host.",
);
}
}
Expand Down
6 changes: 6 additions & 0 deletions src/lib/onboard/sandbox-gpu-direct-proof.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,12 @@ describe("direct sandbox GPU proof", () => {
expect(result.detail).toContain("cuInit(0)=999");
const warnings = warnSpy.mock.calls.map((call) => call[0]).join("\n");
expect(warnings).toContain("/dev/nvmap");
expect(warnings).toContain(
"chmod grants every member of the nvmap owning group write access",
);
expect(warnings.indexOf("chmod grants every member")).toBeLessThan(
warnings.indexOf("sudo chmod g+rw /dev/nvmap"),
);
} finally {
warnSpy.mockRestore();
}
Expand Down
4 changes: 3 additions & 1 deletion src/lib/onboard/sandbox-gpu-preflight.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ export function jetsonGpuProofRemediationLines(): string[] {
return [
"Jetson/Tegra CUDA proof did not pass. CUDA needs access to the Tegra device",
"nodes; confirm the sandbox propagates them and the agent user's groups:",
" ls -l /dev/nvmap /dev/nvhost-* (must be readable by the sandbox)",
" ls -l /dev/nvmap /dev/nvhost-* (nvmap must grant the sandbox group read-write access)",
" warning: chmod grants every member of the nvmap owning group write access",
" sudo chmod g+rw /dev/nvmap (host remediation until udev recreates the device)",
" add the host video/render groups via --group-add when recreating",
"Then recreate the sandbox, or force CPU behavior with NEMOCLAW_SANDBOX_GPU=0.",
];
Expand Down
11 changes: 9 additions & 2 deletions test/e2e/live/jetson-nvmap-gpu.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ exit "$status"`,
progress.phase("confirm nvmap and NVIDIA Docker runtime");
const hostNvmap = await hostShell(
host,
"ls -l /dev/nvmap && stat -c 'gid=%g group=%G' /dev/nvmap",
"ls -l /dev/nvmap && stat -c 'gid=%g group=%G mode=%a' /dev/nvmap",
"phase-0-host-nvmap",
);
expect(hostNvmap.exitCode, resultText(hostNvmap)).toBe(0);
Expand Down Expand Up @@ -264,7 +264,7 @@ exit "$status"`,
expect(sandboxId.exitCode, resultText(sandboxId)).toBe(0);
expectGroupMembership(resultText(sandboxId), hostNvmapGid);

// A6: /dev/nvmap must be mounted/present inside the sandbox.
// A6: /dev/nvmap must be present and read-write for the sandbox user.
const sandboxNvmap = await sandbox.execShell(
SANDBOX_NAME,
trustedSandboxShellScript("ls -l /dev/nvmap"),
Expand All @@ -273,6 +273,13 @@ exit "$status"`,
expect(sandboxNvmap.exitCode, resultText(sandboxNvmap)).toBe(0);
expect(resultText(sandboxNvmap)).toContain("/dev/nvmap");

const sandboxNvmapAccess = await sandbox.execShell(
SANDBOX_NAME,
trustedSandboxShellScript("test -r /dev/nvmap && test -w /dev/nvmap"),
{ artifactName: "phase-3-sandbox-nvmap-access", env: env(), timeoutMs: 60_000 },
);
expect(sandboxNvmapAccess.exitCode, resultText(sandboxNvmapAccess)).toBe(0);

// A7: authoritative CUDA usability proof must succeed, not reproduce
// NvRmMemInitNvmap permission denial / cuInit(0)=999 from #4231.
progress.phase("prove CUDA initialization inside the sandbox");
Expand Down
Loading
Loading