Skip to content

fix(onboard): avoid literal proc wildcard in GPU sandbox policy - #3436

Merged
ericksoa merged 1 commit into
mainfrom
fix/spark-gpu-proc-policy
May 13, 2026
Merged

fix(onboard): avoid literal proc wildcard in GPU sandbox policy#3436
ericksoa merged 1 commit into
mainfrom
fix/spark-gpu-proc-policy

Conversation

@zyang-dev

@zyang-dev zyang-dev commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix direct-GPU sandbox onboarding by removing NemoClaw’s create-time procfs entries that OpenShell should own at runtime. This avoids the literal /proc/self/task/*/comm policy path crash and makes GPU verification compatible with newer OpenShell command validation.

Related Issue

Fixes #3416

Changes

  • Remove /proc and /proc/self/task/*/comm from direct-GPU create policies so OpenShell can apply GPU procfs enrichment.
  • Convert GPU proof shell payloads to single-line commands for OpenShell 0.0.39 compatibility.
  • Fix the proc comm write proof to target the shell’s own /proc/<pid>/task/<tid>/comm entry instead of a short-lived child process.
  • Update onboarding tests for the proc policy behavior and GPU proof command shape.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • make docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: zyang-dev 267119621+zyang-dev@users.noreply.github.com

Summary by CodeRabbit

  • Improvements

    • GPU policy generation now properly excludes /proc filesystem entries based on actual usage patterns rather than static paths.
    • GPU proof command generation includes improved path handling and validation.
  • Tests

    • Expanded test coverage for GPU policy generation with validation of /proc entry removal from policies.
    • Enhanced test assertions for GPU proof command formatting and correctness.

Review Change Stack

Signed-off-by: zyang-dev <267119621+zyang-dev@users.noreply.github.com>
@zyang-dev zyang-dev self-assigned this May 13, 2026
@coderabbitai

coderabbitai Bot commented May 13, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1fe1f7d6-bc60-4ff0-aa6a-0b36bd060284

📥 Commits

Reviewing files that changed from the base of the PR and between d4ae22d and ae131a6.

📒 Files selected for processing (2)
  • src/lib/onboard/initial-policy.ts
  • test/onboard.test.ts

📝 Walkthrough

Walkthrough

GPU policy generation for direct GPU sandboxes now filters /proc filesystem entries owned by OpenShell using a new classification helper. GPU proof command strings were refactored from template literals to command arrays, updating paths from /proc/self/ to /proc/<pid>/task/<tid>/comm format. Tests verify complete /proc removal from policies and updated command path references.

Changes

GPU Policy /proc Filtering and Proof Commands

Layer / File(s) Summary
/proc filtering infrastructure
src/lib/onboard/initial-policy.ts
New PROC_PATH constant and isProcEntryOwnedByOpenShell helper classify filesystem entries. buildDirectGpuPolicyYaml filters both read_only and read_write policies to remove /proc-owned entries instead of fixed /proc checks.
GPU proof command construction and path updates
src/lib/onboard/initial-policy.ts
PROC_COMM_WRITE_PROBE and CUDA_INIT_PROBE refactored from template literals to command arrays joined into one-liners. Comm probe path changes to /proc/$$/task/$$/comm. buildDirectSandboxGpuProofCommands label updated to /proc/<pid>/task/<tid>/comm write.
Test coverage for filtering and commands
test/onboard.test.ts
New test verifies buildDirectGpuPolicyYaml strips /proc. Stricter stale /proc test expects exactly ["/usr"] and ["/tmp"]. Proof commands test verifies /proc/<pid>/task/<tid>/comm paths and no /proc/self/task references.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

OpenShell, fix

Suggested reviewers

  • cv

Poem

🐇 The /proc paths now dance with purpose,
OpenShell's entries banished from the surface,
Arrays replace templates, one-liners take flight,
GPU policies gleaming, filtering done right! ✨

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Linked Issues check ❓ Inconclusive The PR addresses procfs policy issues for GPU sandbox onboarding, but the linked issue #3416 describes sandbox readiness timeout and container creation problems rather than procfs wildcard handling. Clarify how removing /proc entries and converting to single-line commands specifically addresses the 'Create stream exited with code 1' and 180s timeout root causes in #3416.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: fixing GPU sandbox policy by removing literal /proc wildcards to allow OpenShell to properly manage procfs entries.
Out of Scope Changes check ✅ Passed All code changes directly address the PR objective of removing /proc and /proc/self/task/*/comm entries from GPU policies and updating proof commands to single-line format with proper pid/tid references.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/spark-gpu-proc-policy

Comment @coderabbitai help to get the list of available commands and usage tips.

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the narrow GPU procfs policy/proof fix. The current WSL failure is in unrelated Ollama auth-proxy selection tests; this PR is limited to direct-GPU policy generation and proof command shape.

@ericksoa
ericksoa merged commit 98fa96e into main May 13, 2026
28 of 30 checks passed
@ericksoa
ericksoa deleted the fix/spark-gpu-proc-policy branch May 13, 2026 03:44
cv added a commit that referenced this pull request May 15, 2026
## Summary
`NEMOCLAW_SANDBOX_READY_TIMEOUT` has been a recognised env var since
#2849, but no documentation accompanied it —
`docs/reference/commands.md`, `docs/reference/troubleshooting.md`, and
the inference / deployment guides only mention the companion
`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` (added in #1620 and documented at
that time). Operators hitting `Sandbox '<name>' was created but did not
become ready within 180s` have no doc-grep path to the workaround, and
the two timeouts are easy to conflate. This closes the documentation gap
left by #2849.

Originally tried under #3435; closed because that PR mis-framed the docs
as resolving #3344 / #3416 (the root cause of both was the GPU policy
bug fixed in #3436, not a timeout misconfiguration). The docs themselves
still have value as a follow-up to the env-var introductions, so
reopening as a new PR with the correct framing.

## Related Issue
<!-- Not closing any issue; this addresses the doc-gap surfaced while
investigating #3344 and #3416 (both already fixed in code by #3436). -->

## Changes
- `docs/reference/commands.md`: add `NEMOCLAW_SANDBOX_READY_TIMEOUT` and
`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` to the Onboard Timeouts table.
- `docs/reference/troubleshooting.md`: new troubleshooting entry
"Sandbox onboard times out with 'did not become ready within Ns'" that
distinguishes the readiness wait from the inference-probe budget, with a
worked example.
- `docs/inference/use-local-inference.md`: cross-link the two timeouts
from the existing `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` section so readers
of either knob land on the other.
- `docs/deployment/deploy-to-remote-gpu.md`: new "First-Run Readiness
Budget" section calling out DGX Station / cloud-VM /
large-quantised-model conditions that exceed the default and showing how
to raise it.

No code changes — the readiness behaviour is unchanged.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Verification
- [ ] `npx prek run --all-files` passes
- [ ] `npm test` passes
- [ ] Tests added or updated for new or changed behavior
- [ ] No secrets, API keys, or credentials committed
- [ ] Docs updated for user-facing behavior changes
- [x] `make docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a “First-Run Readiness Budget” note for remote GPU hosts
explaining longer initial sandbox build/upload times and advice to
increase NEMOCLAW_SANDBOX_READY_TIMEOUT.
* Clarified that NEMOCLAW_LOCAL_INFERENCE_TIMEOUT applies to
inference-server validation while sandbox readiness uses
NEMOCLAW_SANDBOX_READY_TIMEOUT (default 180s).
* Expanded examples for exporting both timeouts and onboarding timeout
messaging.
* Added troubleshooting guidance and inspection steps when sandbox
readiness timeouts delete partial sandboxes.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3440)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai coderabbitai Bot mentioned this pull request May 16, 2026
15 tasks
@wscurran wscurran added the bug-fix PR fixes a bug or regression label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DGX Station][Install] NemoClaw sandbox not ready in 180 seconds

3 participants