Skip to content

fix: detect cgroup v2 misconfiguration during onboard preflight - #62

Merged
ericksoa merged 1 commit into
NVIDIA:mainfrom
brianwtaylor:fix/cgroup-v2-preflight-check
Mar 17, 2026
Merged

fix: detect cgroup v2 misconfiguration during onboard preflight#62
ericksoa merged 1 commit into
NVIDIA:mainfrom
brianwtaylor:fix/cgroup-v2-preflight-check

Conversation

@brianwtaylor

Copy link
Copy Markdown
Contributor

Summary

Adds a preflight check to nemoclaw onboard (step [1/7]) that detects the cgroup v2 / Docker cgroupns misconfiguration before gateway startup, instead of letting it fail late with a cryptic kubelet error.

  • New module bin/lib/preflight.js with isCgroupV2() and checkCgroupConfig() detection
  • Wired into the existing preflight step in bin/lib/onboard.js between Docker/OpenShell checks and GPU detection
  • On failure: prints the exact error users will see, and points them to nemoclaw setup-spark (which does a safe JSON merge of daemon.json, preserving existing settings)
  • 13 new unit tests in test/preflight.test.js covering all branches

Closes #16

Hardware verification

Tested on two NVIDIA DGX Spark units (aarch64, Ubuntu 24.04.4 LTS, Docker 28.x, cgroup v2):

Machine cgroup daemon.json Detection result Expected
spark0 cgroup2fs missing {ok: false, reason: "...does not exist..."} Correct
spark1 cgroup2fs missing cgroup2fs confirmed, no daemon.json Correct
spark0 (simulated fix) cgroup2fs {"default-cgroupns-mode": "host"} {ok: true} Correct

Both machines are in the exact broken state described in #16. The check catches it and exits with actionable guidance before the gateway ever starts.

Test plan

Automated Tests

node --test test/preflight.test.js
node --test test/*.test.js

Manual verification

  1. On a cgroup v2 host without daemon.json: nemoclaw onboard should fail at step [1/7] with the cgroup error
  2. After running nemoclaw setup-spark: nemoclaw onboard should pass the cgroup check
  3. On a cgroup v1 host or macOS: check is silently skipped (returns {ok: true})

Add a preflight check that catches the #1 onboarding blocker on
Ubuntu 24.04, DGX Spark, and WSL2. When cgroup v2 is active but
Docker's daemon.json lacks "default-cgroupns-mode": "host",
onboarding now fails fast with a clear error and fix instructions
instead of failing late at gateway startup with a cryptic kubelet
error.

Closes #16

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Clean implementation — detection logic is well-separated in preflight.js with dependency injection for testability, error message is actionable (points to nemoclaw setup-spark), and 13 unit tests cover all branches.

Verified: code reads one file (/etc/docker/daemon.json) and runs one safe command (stat -fc %T /sys/fs/cgroup). No shell injection surface, no network calls. macOS/cgroup v1 silently passes.

Hardware verification by the author on two DGX Spark units covers the real detection path that can't be exercised in Docker CI.

@ericksoa
ericksoa merged commit 65dad39 into NVIDIA:main Mar 17, 2026
kagura-agent pushed a commit to kagura-agent/NemoClaw that referenced this pull request Mar 18, 2026
…tibility

On cgroup v2 hosts (Ubuntu 24.04, DGX Spark), k3s-in-Docker requires
cgroupns=host in the Docker daemon config. Without this, kubelet fails
to initialize, the 'openshell' namespace is never created, and gateway
startup times out with:
  'timed out waiting for namespace openshell to exist'

This was previously caught by a preflight check (NVIDIA#62) that was removed
in NVIDIA#248. This commit restores the check in the onboard preflight:

- Detects cgroup v2 by reading /proc/mounts for 'cgroup2'
- Checks /etc/docker/daemon.json for 'default-cgroupns-mode: host'
- On mismatch, exits with a clear error and remediation steps
  (run 'sudo nemoclaw setup-spark' or manually configure daemon.json)
- Shows a success message when cgroup v2 is properly configured

Also improves GPU display for DGX Spark: shows 'unified memory' instead
of 'VRAM' when the Spark GPU is detected (since GB10 uses unified memory
architecture where nvidia-smi cannot report dedicated VRAM).

Fixes NVIDIA#280
jessesanford pushed a commit to jessesanford/NemoClaw that referenced this pull request Mar 24, 2026
…IA#62)

Add a preflight check that catches the NVIDIA#1 onboarding blocker on
Ubuntu 24.04, DGX Spark, and WSL2. When cgroup v2 is active but
Docker's daemon.json lacks "default-cgroupns-mode": "host",
onboarding now fails fast with a clear error and fix instructions
instead of failing late at gateway startup with a cryptic kubelet
error.

Closes NVIDIA#16
mafueee pushed a commit to mafueee/NemoClaw that referenced this pull request Mar 28, 2026
… (!37)

> **🔧 security-fix-agent**

Closes NVIDIA#62

## Security Fix

### Summary
The CONNECT proxy accepted hostnames from clients and connected to whatever IP they resolved to, with no validation against internal address ranges. While the OPA policy is default-deny, a misconfigured or overly permissive policy could allow SSRF to cloud metadata (169.254.169.254), localhost, or RFC1918 services. This fix adds DNS resolution before connecting and rejects any host that resolves to an internal IP.

### Severity Assessment
- **Impact:** Medium — if exploited, could reach cloud metadata (IAM creds), cluster-internal services, or host-local services
- **Exploitability:** Very low — requires OPA policy misconfiguration or DNS rebinding attack
- **Affected components:** `crates/navigator-sandbox/src/proxy.rs` — `handle_tcp_connection`

### Changes Made
- `crates/navigator-sandbox/src/proxy.rs`: Added `is_internal_ip()` helper that checks IPv4 loopback/private/link-local, IPv6 loopback/link-local, and IPv4-mapped IPv6. Added `resolve_and_reject_internal()` that resolves DNS and rejects internal IPs. Inserted check between OPA allow and `TcpStream::connect`, with control plane endpoints exempt.
- `architecture/security-policy.md`: Added SSRF Protection section with blocked ranges table and flow diagram
- `architecture/sandbox.md`: Updated proxy connection flow diagram and added SSRF protection subsection
- `architecture/README.md`: Added internal IP rejection step to proxy description

### Tests Added
- **Unit:** 17 tests in `proxy::tests` — covers IPv4 loopback/private/link-local, IPv6 loopback/link-local, IPv4-mapped IPv6, public IPs, DNS resolution of localhost/127.0.0.1/169.254.169.254, and DNS failure handling
- **Integration/E2E:** N/A — the proxy runs inside a Linux network namespace; unit tests for IP checking and DNS resolution cover the security boundary

### Documentation Updated
- `architecture/security-policy.md`: New SSRF Protection section with blocked IP ranges and Mermaid flowchart
- `architecture/sandbox.md`: Updated proxy flow diagram and added SSRF protection subsection
- `architecture/README.md`: Added step 4 to proxy description

### Verification
All 85 sandbox tests pass including 17 new proxy SSRF tests. Pre-commit (fmt, clippy, full test suite) passes clean with zero warnings.
mafueee pushed a commit to mafueee/NemoClaw that referenced this pull request Mar 28, 2026
…VIDIA#62)

TLS certificates are always resolved automatically from cluster metadata,
making the explicit CLI flags unnecessary. The TlsOptions struct and
auto-resolution logic remain intact for programmatic and test use.
@wscurran wscurran added the bug-fix PR fixes a bug or regression label Jun 8, 2026
@derekhsu

Copy link
Copy Markdown

Code Review — PR #62 (post-merge)

Reviewed the merged diff (fc31ae8): new bin/lib/preflight.js + wiring in bin/lib/onboard.js + test/preflight.test.js. This is a retrospective review, so issues are framed as potential follow-ups rather than merge blockers.

Strengths

  • Clear, actionable failure path. The error printed in onboard.js shows the exact kubelet error users hit (openat2 /sys/fs/cgroup/kubepods/pids.max) and points straight to nemoclaw setup-spark. This directly addresses the "fails late with a cryptic error" problem from Detect and remediate the cgroup v2 / "Kubernetes namespace not ready" onboarding blocker on Ubuntu 24.04 #16.
  • Testable design. checkCgroupConfig(opts) injects both cgroupV2 and daemonPath, so the unit tests exercise real file I/O against temp daemon.json files instead of mocks. All branches are covered (v1 skip, v2+host, missing file, missing key, wrong value "private", invalid JSON, extra keys).
  • Correct platform guard. isCgroupV2() short-circuits to false on non-Linux before ever invoking stat, so macOS never depends on command failure as a fallback.
  • Safe parse. readDaemonJson() swallows both missing-file and parse errors, so a malformed daemon.json can't crash onboarding.

Issues

Important (Should Fix — follow-up)

  1. stat -fc %T is fragile across coreutils implementationsbin/lib/preflight.js isCgroupV2()

    • stat -fc %T /sys/fs/cgroup relies on GNU coreutils emitting the literal cgroup2fs. uutils/Rust coreutils (shipping on some minimal/Ubuntu images) can emit UNKNOWN (0x63677270) instead, which makes isCgroupV2() return false and silently skips the check — so the user still hits the late kubelet failure this PR was meant to prevent.
    • Fix: fall back to a filesystem signal that doesn't depend on stat output formatting, e.g. existence of /sys/fs/cgroup/cgroup.controllers (cgroup v2-only) or parsing /proc/mounts.
  2. Detection only inspects daemon.json → false-positive process.exit(1)bin/lib/preflight.js checkCgroupConfig()

    • default-cgroupns-mode=host can also be set via a dockerd CLI flag or a systemd drop-in (/etc/systemd/system/docker.service.d/*.conf). A host correctly configured that way still has no (or a different) daemon.json, so the check fails and onboard.js calls process.exit(1), hard-blocking a working setup.
    • Fix: before failing, also consult the live daemon (docker info's cgroup/cgroupns fields) or the effective ExecStart from systemctl show docker.

Minor (Nice to Have)

  1. reason conflates "missing file" with "invalid JSON"bin/lib/preflight.js

    • readDaemonJson() returns null for both cases, so the reason text is always "…does not exist or is not valid JSON". A user with a typo'd daemon.json is told it "does not exist," which misleads debugging. Consider returning a small struct ({ exists, parsed }) so the message can distinguish the two. (The test only asserts substring "not valid JSON", which happens to be present in the combined message, so this gap isn't caught.)
  2. No non-interactive short-circuitbin/lib/onboard.js

    • The multi-line guidance is fine for a TTY but is noisy in CI/non-interactive runs where the user can't act on it. A shorter message in non-interactive mode would be cleaner. (Minor.)

Important context for maintainers (not a defect in this PR)

This preflight no longer exists in main in its merged form. It was reimplemented in TypeScript at src/lib/onboard/preflight.ts, and the enforcement is now deliberately disabled: requiresHostCgroupnsFix: false is hardcoded with the comment "Current OpenShell sets host cgroupns on its own cluster container." (preflight.ts:584-585). So the cgroup data is still collected but never blocks onboarding. If that comment is accurate, issues #1/#2 above are moot going forward; if OpenShell's behavior regresses, the same portability/false-positive concerns will resurface in the TS version. Worth confirming the disable was intentional and documenting why.

Assessment

Ready to merge? Already merged — would have been With fixes. The core implementation is clean, well-tested, and solves the stated problem; the two Important items (coreutils portability and daemon.json-only detection) are the parts most likely to cause real-world false negatives/positives and are the best follow-up candidates.

🤖 Review generated with Devin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detect and remediate the cgroup v2 / "Kubernetes namespace not ready" onboarding blocker on Ubuntu 24.04

4 participants