Skip to content

fix(install): simplify DGX Spark setup, remove cgroup v2 workaround #1368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 3 commits into from
Apr 3, 2026
Merged

fix(install): simplify DGX Spark setup, remove cgroup v2 workaround #1368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cb07f85
fix(install): simplify DGX Spark setup, remove obsolete cgroup v2 w…
zyang-dev Mar 30, 2026
f1dc3db
fix: clean up setup-spark.sh from rebase
zyang-dev Apr 2, 2026
814bcd5
Merge branch 'main' into fix/spark-install-remove-cgroup-workaround
kjw3 Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 10 additions & 81 deletions scripts/setup-spark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,16 @@
#
# NemoClaw setup for DGX Spark devices.
#
# Spark ships Ubuntu 24.04 (cgroup v2) + Docker 28.x but no k3s.
# OpenShell's gateway starts k3s inside a Docker container, which
# needs cgroup host namespace access. This script configures Docker
# for that.
# Ensures the current user is in the docker group so NemoClaw can
# manage containers without sudo.
#
# Usage:
# sudo nemoclaw setup-spark
# # or directly:
# sudo bash scripts/setup-spark.sh
# # or via curl:
# curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash
#
# What it does:
# 1. Adds current user to docker group (avoids sudo for everything else)
# 2. Configures Docker daemon for cgroupns=host (k3s-in-Docker on cgroup v2)
# 3. Restarts Docker

set -euo pipefail

Expand Down Expand Up @@ -59,82 +55,15 @@ if [ -n "$REAL_USER" ]; then
else
info "Adding '$REAL_USER' to docker group..."
usermod -aG docker "$REAL_USER"
info "Added. Group will take effect on next login (or use 'newgrp docker')."
DOCKER_GROUP_ADDED=true
fi
fi

# ── 2. Docker cgroup namespace ────────────────────────────────────
#
# Spark runs cgroup v2 (Ubuntu 24.04). OpenShell's gateway embeds
# k3s in a Docker container, which needs --cgroupns=host to manage
# cgroup hierarchies. Without this, kubelet fails with:
# "openat2 /sys/fs/cgroup/kubepods/pids.max: no"
#
# Setting default-cgroupns-mode=host in daemon.json makes all
# containers use the host cgroup namespace. This is safe — it's
# the Docker default on cgroup v1 hosts anyway.

DAEMON_JSON="/etc/docker/daemon.json"
NEEDS_RESTART=false
# ── 2. Next steps ─────────────────────────────────────────────────

if [ -f "$DAEMON_JSON" ]; then
# Check if already configured
if grep -q '"default-cgroupns-mode"' "$DAEMON_JSON" 2>/dev/null; then
CURRENT_MODE=$(python3 -c "import json; print(json.load(open('$DAEMON_JSON')).get('default-cgroupns-mode',''))" 2>/dev/null || echo "")
if [ "$CURRENT_MODE" = "host" ]; then
info "Docker daemon already configured for cgroupns=host"
else
info "Updating Docker daemon cgroupns mode to 'host'..."
python3 -c "
import json
with open('$DAEMON_JSON') as f:
d = json.load(f)
d['default-cgroupns-mode'] = 'host'
with open('$DAEMON_JSON', 'w') as f:
json.dump(d, f, indent=2)
"
NEEDS_RESTART=true
fi
else
info "Adding cgroupns=host to Docker daemon config..."
python3 -c "
import json
try:
with open('$DAEMON_JSON') as f:
d = json.load(f)
except:
d = {}
d['default-cgroupns-mode'] = 'host'
with open('$DAEMON_JSON', 'w') as f:
json.dump(d, f, indent=2)
"
NEEDS_RESTART=true
fi
echo ""
if [ "${DOCKER_GROUP_ADDED:-}" = true ]; then
warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing."
else
info "Creating Docker daemon config with cgroupns=host..."
mkdir -p "$(dirname "$DAEMON_JSON")"
echo '{ "default-cgroupns-mode": "host" }' >"$DAEMON_JSON"
NEEDS_RESTART=true
info "DGX Spark Docker configuration complete."
fi

# ── 3. Restart Docker if needed ───────────────────────────────────

if [ "$NEEDS_RESTART" = true ]; then
info "Restarting Docker daemon..."
systemctl restart docker
# Wait for Docker to be ready
for i in 1 2 3 4 5 6 7 8 9 10; do
if docker info >/dev/null 2>&1; then
break
fi
[ "$i" -eq 10 ] && fail "Docker didn't come back after restart. Check 'systemctl status docker'."
sleep 2
done
info "Docker restarted with cgroupns=host"
fi

# ── 4. Run normal setup ──────────────────────────────────────────

echo ""
info "DGX Spark Docker configuration complete."
info ""
56 changes: 14 additions & 42 deletions spark-install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,16 @@ Before starting, make sure you have:

- **Docker** (pre-installed on DGX Spark, v28.x/29.x)
- **Node.js 22** (installed automatically by the NemoClaw installer)
- **OpenShell CLI** (must be installed separately before running NemoClaw — see the Quick Start below)
- **OpenShell CLI** (installed automatically by the NemoClaw installer)
- **API key** (cloud inference only) — the onboarding wizard prompts for a provider and key during setup. For example, an NVIDIA API key from [build.nvidia.com](https://build.nvidia.com) for NVIDIA Endpoints, or an OpenAI, Anthropic, or Gemini key for those providers. **If you plan to use local inference with Ollama instead, no API key is needed** — see [Local Inference with Ollama](#local-inference-with-ollama) to set up Ollama before installing NemoClaw.

## Quick Start

```bash
# Install OpenShell:
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

# Clone NemoClaw:
git clone https://github.com/NVIDIA/NemoClaw.git
cd NemoClaw

# Spark-specific setup (fixes cgroup v2 and Docker permissions — see Troubleshooting for details)
sudo ./scripts/setup-spark.sh
# Spark-specific setup (requires sudo)
curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash

# Install NemoClaw:
./install.sh

# Alternatively, you can use the hosted install script:
# Install NemoClaw
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

Expand Down Expand Up @@ -124,8 +114,6 @@ If NemoClaw is **already installed** with a cloud provider and you want to switc

```bash
nemoclaw uninstall

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

Expand Down Expand Up @@ -158,7 +146,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi

| Issue | Status | Workaround |
|-------|--------|------------|
| cgroup v2 kills k3s in Docker | Fixed in `setup-spark` | `daemon.json` cgroupns=host |
| cgroup v2 kills k3s in Docker | Fixed in recent OpenShell versions | OpenShell sets `cgroupns=host` on the gateway container directly |
| Docker permission denied | Fixed in `setup-spark` | `usermod -aG docker` |
| CoreDNS CrashLoop after setup | Fixed in `fix-coredns.sh` | Uses container gateway IP, not 127.0.0.11 |
| Image pull failure (k3s can't find built image) | OpenShell bug | `openshell gateway destroy && openshell gateway start`, re-run setup |
Expand All @@ -169,27 +157,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi

### Manual Setup (if setup-spark doesn't work)

If `setup-spark.sh` fails, you can apply the fixes it performs by hand:

#### Fix Docker cgroup namespace

```bash
# Check if you're on cgroup v2
stat -fc %T /sys/fs/cgroup/
# Expected: cgroup2fs

# Add cgroupns=host to Docker daemon config
sudo python3 -c "
import json, os
path = '/etc/docker/daemon.json'
d = json.load(open(path)) if os.path.exists(path) else {}
d['default-cgroupns-mode'] = 'host'
json.dump(d, open(path, 'w'), indent=2)
"

# Restart Docker
sudo systemctl restart docker
```
If `setup-spark.sh` fails, you can apply the fix it performs by hand:

#### Fix Docker permissions

Expand Down Expand Up @@ -230,23 +198,27 @@ Error in the hyper legacy client: client error (Connect)
**Cause**: Your user isn't in the `docker` group.
**Fix**: `setup-spark` runs `usermod -aG docker $USER`. You may need to log out and back in (or `newgrp docker`) for it to take effect.

#### cgroup v2 incompatibility
#### cgroup v2 incompatibility (resolved)

```text
K8s namespace not ready
openat2 /sys/fs/cgroup/kubepods/pids.max: no
Failed to start ContainerManager: failed to initialize top level QOS containers
```

**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist. The fix is `--cgroupns=host` on the container, but OpenShell doesn't expose that flag.
**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist without host cgroup namespace access.

**Fix**: `setup-spark` sets `"default-cgroupns-mode": "host"` in `/etc/docker/daemon.json` and restarts Docker. This makes all containers use the host cgroup namespace, which is what k3s needs.
**Fix**: Recent OpenShell versions set `cgroupns=host` on the gateway container directly ([OpenShell PR #329](https://github.com/NVIDIA/OpenShell/pull/329)). No `daemon.json` workaround is needed. If you are on an older OpenShell version, upgrade with:

```bash
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
```

### Architecture

```text
DGX Spark (Ubuntu 24.04, aarch64, cgroup v2, 128 GB unified memory)
└── Docker (28.x/29.x, cgroupns=host)
└── Docker (28.x/29.x)
└── OpenShell gateway container
└── k3s (embedded)
└── nemoclaw sandbox pod
Expand Down
Loading