Skip to content

fix(install): simplify DGX Spark setup, remove cgroup v2 workaround #1394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zyang-dev wants to merge 3 commits into from
Closed

fix(install): simplify DGX Spark setup, remove cgroup v2 workaround #1394

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0776f97
fix(install): simplify DGX Spark setup, remove cgroup v2 workaround
zyang-dev Apr 3, 2026
7cbee93
Merge branch 'main' into fix/spark-remove-cgroup-workaround
kjw3 Apr 3, 2026
b5f30fa
Merge branch 'main' into fix/spark-remove-cgroup-workaround
cv Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 10 additions & 81 deletions scripts/setup-spark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,16 @@
#
# NemoClaw setup for DGX Spark devices.
#
# Spark ships Ubuntu 24.04 (cgroup v2) + Docker 28.x but no k3s.
# OpenShell's gateway starts k3s inside a Docker container, which
# needs cgroup host namespace access. This script configures Docker
# for that.
# Ensures the current user is in the docker group so NemoClaw can
# manage containers without sudo.
#
# Usage:
# sudo nemoclaw setup-spark
# # or directly:
# sudo bash scripts/setup-spark.sh
# # or via curl:
# curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash
#
# What it does:
# 1. Adds current user to docker group (avoids sudo for everything else)
# 2. Configures Docker daemon for cgroupns=host (k3s-in-Docker on cgroup v2)
# 3. Restarts Docker

set -euo pipefail

Expand Down Expand Up @@ -59,82 +55,15 @@ if [ -n "$REAL_USER" ]; then
else
info "Adding '$REAL_USER' to docker group..."
usermod -aG docker "$REAL_USER"
info "Added. Group will take effect on next login (or use 'newgrp docker')."
DOCKER_GROUP_ADDED=true
fi
fi

# ── 2. Docker cgroup namespace ────────────────────────────────────
#
# Spark runs cgroup v2 (Ubuntu 24.04). OpenShell's gateway embeds
# k3s in a Docker container, which needs --cgroupns=host to manage
# cgroup hierarchies. Without this, kubelet fails with:
# "openat2 /sys/fs/cgroup/kubepods/pids.max: no"
#
# Setting default-cgroupns-mode=host in daemon.json makes all
# containers use the host cgroup namespace. This is safe — it's
# the Docker default on cgroup v1 hosts anyway.

DAEMON_JSON="/etc/docker/daemon.json"
NEEDS_RESTART=false
# ── 2. Next steps ─────────────────────────────────────────────────

if [ -f "$DAEMON_JSON" ]; then
# Check if already configured
if grep -q '"default-cgroupns-mode"' "$DAEMON_JSON" 2>/dev/null; then
CURRENT_MODE=$(python3 -c "import json; print(json.load(open('$DAEMON_JSON')).get('default-cgroupns-mode',''))" 2>/dev/null || echo "")
if [ "$CURRENT_MODE" = "host" ]; then
info "Docker daemon already configured for cgroupns=host"
else
info "Updating Docker daemon cgroupns mode to 'host'..."
python3 -c "
import json
with open('$DAEMON_JSON') as f:
d = json.load(f)
d['default-cgroupns-mode'] = 'host'
with open('$DAEMON_JSON', 'w') as f:
json.dump(d, f, indent=2)
"
NEEDS_RESTART=true
fi
else
info "Adding cgroupns=host to Docker daemon config..."
python3 -c "
import json
try:
with open('$DAEMON_JSON') as f:
d = json.load(f)
except:
d = {}
d['default-cgroupns-mode'] = 'host'
with open('$DAEMON_JSON', 'w') as f:
json.dump(d, f, indent=2)
"
NEEDS_RESTART=true
fi
echo ""
if [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing."
else
info "Creating Docker daemon config with cgroupns=host..."
mkdir -p "$(dirname "$DAEMON_JSON")"
echo '{ "default-cgroupns-mode": "host" }' >"$DAEMON_JSON"
NEEDS_RESTART=true
info "DGX Spark Docker configuration complete."
fi
Comment on lines +64 to 69

@coderabbitai coderabbitai Bot Apr 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Avoid reporting full success when no non-root user was detected

If the warning path at Line 45 is hit, Line 68 still reports completion even though no docker-group configuration was applied. That can create a false-success path before install.

Suggested patch 
 echo ""
-if [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
+if [ -z "$REAL_USER" ]; then
+  warn "Setup finished, but no non-root user was detected; docker group was not configured."
+elif [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
   warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing."
 else
   info "DGX Spark Docker configuration complete."
 fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo ""
if [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing."
else
info "Creating Docker daemon config with cgroupns=host..."
mkdir -p "$(dirname "$DAEMON_JSON")"
echo '{ "default-cgroupns-mode": "host" }' >"$DAEMON_JSON"
NEEDS_RESTART=true
info "DGX Spark Docker configuration complete."
fi
echo ""
if [ -z "$REAL_USER" ]; then
warn "Setup finished, but no non-root user was detected; docker group was not configured."
elif [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing."
else
info "DGX Spark Docker configuration complete."
fi
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@scripts/setup-spark.sh` around lines 64 - 69, The script prints a final
success message even when the earlier warn path for DOCKER_GROUP_ADDED was
triggered; update the tailing conditional so the info("DGX Spark Docker
configuration complete.") is only emitted when DOCKER_GROUP_ADDED is not true
(i.e., when no warning was previously issued/when the docker-group was already
present or configuration actually completed), and ensure the same
DOCKER_GROUP_ADDED flag used earlier is checked here so the warn branch cannot
fall through to the info branch (reference DOCKER_GROUP_ADDED, warn, and info).


# ── 3. Restart Docker if needed ───────────────────────────────────

if [ "$NEEDS_RESTART" = true ]; then
info "Restarting Docker daemon..."
systemctl restart docker
# Wait for Docker to be ready
for i in 1 2 3 4 5 6 7 8 9 10; do
if docker info >/dev/null 2>&1; then
break
fi
[ "$i" -eq 10 ] && fail "Docker didn't come back after restart. Check 'systemctl status docker'."
sleep 2
done
info "Docker restarted with cgroupns=host"
fi

# ── 4. Run normal setup ──────────────────────────────────────────

echo ""
info "DGX Spark Docker configuration complete."
info ""
56 changes: 14 additions & 42 deletions spark-install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,16 @@ Before starting, make sure you have:

- **Docker** (pre-installed on DGX Spark, v28.x/29.x)
- **Node.js 22** (installed automatically by the NemoClaw installer)
- **OpenShell CLI** (must be installed separately before running NemoClaw — see the Quick Start below)
- **OpenShell CLI** (installed automatically by the NemoClaw installer)
- **API key** (cloud inference only) — the onboarding wizard prompts for a provider and key during setup. For example, an NVIDIA API key from [build.nvidia.com](https://build.nvidia.com) for NVIDIA Endpoints, or an OpenAI, Anthropic, or Gemini key for those providers. **If you plan to use local inference with Ollama instead, no API key is needed** — see [Local Inference with Ollama](#local-inference-with-ollama) to set up Ollama before installing NemoClaw.

## Quick Start

```bash
# Install OpenShell:
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

# Clone NemoClaw:
git clone https://github.com/NVIDIA/NemoClaw.git
cd NemoClaw

# Spark-specific setup (fixes cgroup v2 and Docker permissions — see Troubleshooting for details)
sudo ./scripts/setup-spark.sh
# Spark-specific setup (requires sudo)
curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash

# Install NemoClaw:
./install.sh

# Alternatively, you can use the hosted install script:
# Install NemoClaw
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```
Comment on lines 18 to 24

@coderabbitai coderabbitai Bot Apr 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add an explicit shell-refresh note between Quick Start commands

After Line 20, docker group membership may not be active in the current shell yet. Running Line 23 immediately (especially via full-block paste) can fail with Docker permission errors.

Suggested doc tweak (keeps Quick Start to the same two commands) 
 ```bash
 # Spark-specific setup (requires sudo)
 curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash
-
-# Install NemoClaw
-curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

+If setup-spark.sh reports your user was just added to the docker group, open a new terminal (or run newgrp docker) before continuing.
+
+bash +# Install NemoClaw +curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash +

</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@spark-install.md` around lines 18 - 24, The Quick Start sequence runs
setup-spark.sh then immediately runs the NemoClaw installer, but if
setup-spark.sh added your user to the docker group the new group membership may
not be active in the current shell causing Docker permission errors; update the
docs around the setup-spark.sh step to add a short note instructing the user to
open a new terminal or run newgrp docker (or otherwise refresh their shell) if
the script reports they were added to the docker group before running the
NemoClaw install curl command.


Expand Down Expand Up @@ -124,8 +114,6 @@ If NemoClaw is **already installed** with a cloud provider and you want to switc

```bash
nemoclaw uninstall

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

Expand Down Expand Up @@ -158,7 +146,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi

| Issue | Status | Workaround |
|-------|--------|------------|
| cgroup v2 kills k3s in Docker | Fixed in `setup-spark` | `daemon.json` cgroupns=host |
| cgroup v2 kills k3s in Docker | Fixed in recent OpenShell versions | OpenShell sets `cgroupns=host` on the gateway container directly |
| Docker permission denied | Fixed in `setup-spark` | `usermod -aG docker` |
| CoreDNS CrashLoop after setup | Fixed in `fix-coredns.sh` | Uses container gateway IP, not 127.0.0.11 |
| Image pull failure (k3s can't find built image) | OpenShell bug | `openshell gateway destroy && openshell gateway start`, re-run setup |
Expand All @@ -169,27 +157,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi

### Manual Setup (if setup-spark doesn't work)

If `setup-spark.sh` fails, you can apply the fixes it performs by hand:

#### Fix Docker cgroup namespace

```bash
# Check if you're on cgroup v2
stat -fc %T /sys/fs/cgroup/
# Expected: cgroup2fs

# Add cgroupns=host to Docker daemon config
sudo python3 -c "
import json, os
path = '/etc/docker/daemon.json'
d = json.load(open(path)) if os.path.exists(path) else {}
d['default-cgroupns-mode'] = 'host'
json.dump(d, open(path, 'w'), indent=2)
"

# Restart Docker
sudo systemctl restart docker
```
If `setup-spark.sh` fails, you can apply the fix it performs by hand:

#### Fix Docker permissions

Expand Down Expand Up @@ -230,23 +198,27 @@ Error in the hyper legacy client: client error (Connect)
**Cause**: Your user isn't in the `docker` group.
**Fix**: `setup-spark` runs `usermod -aG docker $USER`. You may need to log out and back in (or `newgrp docker`) for it to take effect.

#### cgroup v2 incompatibility
#### cgroup v2 incompatibility (resolved)

```text
K8s namespace not ready
openat2 /sys/fs/cgroup/kubepods/pids.max: no
Failed to start ContainerManager: failed to initialize top level QOS containers
```

**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist. The fix is `--cgroupns=host` on the container, but OpenShell doesn't expose that flag.
**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist without host cgroup namespace access.

**Fix**: `setup-spark` sets `"default-cgroupns-mode": "host"` in `/etc/docker/daemon.json` and restarts Docker. This makes all containers use the host cgroup namespace, which is what k3s needs.
**Fix**: Recent OpenShell versions set `cgroupns=host` on the gateway container directly ([OpenShell PR #329](https://github.com/NVIDIA/OpenShell/pull/329)). No `daemon.json` workaround is needed. If you are on an older OpenShell version, upgrade with:

```bash
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
```

### Architecture

```text
DGX Spark (Ubuntu 24.04, aarch64, cgroup v2, 128 GB unified memory)
└── Docker (28.x/29.x, cgroupns=host)
└── Docker (28.x/29.x)
└── OpenShell gateway container
└── k3s (embedded)
└── nemoclaw sandbox pod
Expand Down
Loading