diff --git a/scripts/setup-spark.sh b/scripts/setup-spark.sh index 2cbcf6e556b..6e420425897 100755 --- a/scripts/setup-spark.sh +++ b/scripts/setup-spark.sh @@ -4,20 +4,16 @@ # # NemoClaw setup for DGX Spark devices. # -# Spark ships Ubuntu 24.04 (cgroup v2) + Docker 28.x but no k3s. -# OpenShell's gateway starts k3s inside a Docker container, which -# needs cgroup host namespace access. This script configures Docker -# for that. +# Ensures the current user is in the docker group so NemoClaw can +# manage containers without sudo. # # Usage: -# sudo nemoclaw setup-spark -# # or directly: # sudo bash scripts/setup-spark.sh +# # or via curl: +# curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash # # What it does: # 1. Adds current user to docker group (avoids sudo for everything else) -# 2. Configures Docker daemon for cgroupns=host (k3s-in-Docker on cgroup v2) -# 3. Restarts Docker set -euo pipefail @@ -59,82 +55,15 @@ if [ -n "$REAL_USER" ]; then else info "Adding '$REAL_USER' to docker group..." usermod -aG docker "$REAL_USER" - info "Added. Group will take effect on next login (or use 'newgrp docker')." + DOCKER_GROUP_ADDED=true fi fi -# ── 2. Docker cgroup namespace ──────────────────────────────────── -# -# Spark runs cgroup v2 (Ubuntu 24.04). OpenShell's gateway embeds -# k3s in a Docker container, which needs --cgroupns=host to manage -# cgroup hierarchies. Without this, kubelet fails with: -# "openat2 /sys/fs/cgroup/kubepods/pids.max: no" -# -# Setting default-cgroupns-mode=host in daemon.json makes all -# containers use the host cgroup namespace. This is safe — it's -# the Docker default on cgroup v1 hosts anyway. - -DAEMON_JSON="/etc/docker/daemon.json" -NEEDS_RESTART=false +# ── 2. Next steps ───────────────────────────────────────────────── -if [ -f "$DAEMON_JSON" ]; then - # Check if already configured - if grep -q '"default-cgroupns-mode"' "$DAEMON_JSON" 2>/dev/null; then - CURRENT_MODE=$(python3 -c "import json; print(json.load(open('$DAEMON_JSON')).get('default-cgroupns-mode',''))" 2>/dev/null || echo "") - if [ "$CURRENT_MODE" = "host" ]; then - info "Docker daemon already configured for cgroupns=host" - else - info "Updating Docker daemon cgroupns mode to 'host'..." - python3 -c " -import json -with open('$DAEMON_JSON') as f: - d = json.load(f) -d['default-cgroupns-mode'] = 'host' -with open('$DAEMON_JSON', 'w') as f: - json.dump(d, f, indent=2) -" - NEEDS_RESTART=true - fi - else - info "Adding cgroupns=host to Docker daemon config..." - python3 -c " -import json -try: - with open('$DAEMON_JSON') as f: - d = json.load(f) -except: - d = {} -d['default-cgroupns-mode'] = 'host' -with open('$DAEMON_JSON', 'w') as f: - json.dump(d, f, indent=2) -" - NEEDS_RESTART=true - fi +echo "" +if [ "${DOCKER_GROUP_ADDED:-}" = true ]; then + warn "Docker group was just added. You must open a new terminal (or run 'newgrp docker') before continuing." else - info "Creating Docker daemon config with cgroupns=host..." - mkdir -p "$(dirname "$DAEMON_JSON")" - echo '{ "default-cgroupns-mode": "host" }' >"$DAEMON_JSON" - NEEDS_RESTART=true + info "DGX Spark Docker configuration complete." fi - -# ── 3. Restart Docker if needed ─────────────────────────────────── - -if [ "$NEEDS_RESTART" = true ]; then - info "Restarting Docker daemon..." - systemctl restart docker - # Wait for Docker to be ready - for i in 1 2 3 4 5 6 7 8 9 10; do - if docker info >/dev/null 2>&1; then - break - fi - [ "$i" -eq 10 ] && fail "Docker didn't come back after restart. Check 'systemctl status docker'." - sleep 2 - done - info "Docker restarted with cgroupns=host" -fi - -# ── 4. Run normal setup ────────────────────────────────────────── - -echo "" -info "DGX Spark Docker configuration complete." -info "" diff --git a/spark-install.md b/spark-install.md index 34a6702bd0a..94cf265bfa5 100644 --- a/spark-install.md +++ b/spark-install.md @@ -10,26 +10,16 @@ Before starting, make sure you have: - **Docker** (pre-installed on DGX Spark, v28.x/29.x) - **Node.js 22** (installed automatically by the NemoClaw installer) -- **OpenShell CLI** (must be installed separately before running NemoClaw — see the Quick Start below) +- **OpenShell CLI** (installed automatically by the NemoClaw installer) - **API key** (cloud inference only) — the onboarding wizard prompts for a provider and key during setup. For example, an NVIDIA API key from [build.nvidia.com](https://build.nvidia.com) for NVIDIA Endpoints, or an OpenAI, Anthropic, or Gemini key for those providers. **If you plan to use local inference with Ollama instead, no API key is needed** — see [Local Inference with Ollama](#local-inference-with-ollama) to set up Ollama before installing NemoClaw. ## Quick Start ```bash -# Install OpenShell: -curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh - -# Clone NemoClaw: -git clone https://github.com/NVIDIA/NemoClaw.git -cd NemoClaw - -# Spark-specific setup (fixes cgroup v2 and Docker permissions — see Troubleshooting for details) -sudo ./scripts/setup-spark.sh +# Spark-specific setup (requires sudo) +curl -fsSL https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/scripts/setup-spark.sh | sudo bash -# Install NemoClaw: -./install.sh - -# Alternatively, you can use the hosted install script: +# Install NemoClaw curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash ``` @@ -124,8 +114,6 @@ If NemoClaw is **already installed** with a cloud provider and you want to switc ```bash nemoclaw uninstall - -curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash ``` @@ -158,7 +146,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi | Issue | Status | Workaround | |-------|--------|------------| -| cgroup v2 kills k3s in Docker | Fixed in `setup-spark` | `daemon.json` cgroupns=host | +| cgroup v2 kills k3s in Docker | Fixed in recent OpenShell versions | OpenShell sets `cgroupns=host` on the gateway container directly | | Docker permission denied | Fixed in `setup-spark` | `usermod -aG docker` | | CoreDNS CrashLoop after setup | Fixed in `fix-coredns.sh` | Uses container gateway IP, not 127.0.0.11 | | Image pull failure (k3s can't find built image) | OpenShell bug | `openshell gateway destroy && openshell gateway start`, re-run setup | @@ -169,27 +157,7 @@ openclaw agent --agent main --local -m "Which model and GPU are in use?" --sessi ### Manual Setup (if setup-spark doesn't work) -If `setup-spark.sh` fails, you can apply the fixes it performs by hand: - -#### Fix Docker cgroup namespace - -```bash -# Check if you're on cgroup v2 -stat -fc %T /sys/fs/cgroup/ -# Expected: cgroup2fs - -# Add cgroupns=host to Docker daemon config -sudo python3 -c " -import json, os -path = '/etc/docker/daemon.json' -d = json.load(open(path)) if os.path.exists(path) else {} -d['default-cgroupns-mode'] = 'host' -json.dump(d, open(path, 'w'), indent=2) -" - -# Restart Docker -sudo systemctl restart docker -``` +If `setup-spark.sh` fails, you can apply the fix it performs by hand: #### Fix Docker permissions @@ -230,7 +198,7 @@ Error in the hyper legacy client: client error (Connect) **Cause**: Your user isn't in the `docker` group. **Fix**: `setup-spark` runs `usermod -aG docker $USER`. You may need to log out and back in (or `newgrp docker`) for it to take effect. -#### cgroup v2 incompatibility +#### cgroup v2 incompatibility (resolved) ```text K8s namespace not ready @@ -238,15 +206,19 @@ openat2 /sys/fs/cgroup/kubepods/pids.max: no Failed to start ContainerManager: failed to initialize top level QOS containers ``` -**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist. The fix is `--cgroupns=host` on the container, but OpenShell doesn't expose that flag. +**Cause**: Spark runs cgroup v2 (Ubuntu 24.04 default). OpenShell's gateway container starts k3s, which tries to create cgroup v1-style paths that don't exist without host cgroup namespace access. -**Fix**: `setup-spark` sets `"default-cgroupns-mode": "host"` in `/etc/docker/daemon.json` and restarts Docker. This makes all containers use the host cgroup namespace, which is what k3s needs. +**Fix**: Recent OpenShell versions set `cgroupns=host` on the gateway container directly ([OpenShell PR #329](https://github.com/NVIDIA/OpenShell/pull/329)). No `daemon.json` workaround is needed. If you are on an older OpenShell version, upgrade with: + +```bash +curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh +``` ### Architecture ```text DGX Spark (Ubuntu 24.04, aarch64, cgroup v2, 128 GB unified memory) - └── Docker (28.x/29.x, cgroupns=host) + └── Docker (28.x/29.x) └── OpenShell gateway container └── k3s (embedded) └── nemoclaw sandbox pod