Skip to content

docs(policy): fix OpenShell policy round-trip examples - #5880

Merged
cv merged 5 commits into
mainfrom
fix/5764-policy-doc-roundtrip
Jun 26, 2026
Merged

docs(policy): fix OpenShell policy round-trip examples#5880
cv merged 5 commits into
mainfrom
fix/5764-policy-doc-roundtrip

Conversation

@miyoungc

@miyoungc miyoungc commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Update the raw OpenShell policy replacement examples so they match NemoClaw's pinned OpenShell 0.0.44 command surface.
The examples now export the full live policy with openshell policy get <name> --full, strip the OpenShell metadata header before editing, and apply the file back with the name-first policy set syntax.

Related Issue

Fixes #5764

Changes

  • Update the customize network policy guide to use the pinned-compatible openshell policy get <name> --full export before raw policy replacement.
  • Add a fail-closed metadata-header stripping step before writing current-policy.yaml for manual edits.
  • Align related policy snippets in integration examples, the CLI selection guide, and the network policy reference with OpenShell 0.0.44 syntax.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification: doc-only correction verified against the OpenShell 0.0.44 CLI parser and published policy docs.
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: docs-only policy command examples checked against OpenShell 0.0.44 PolicyCommands::Set/Get; no runtime policy enforcement code changed.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — npm run docs passed, but Fern reports an existing light-mode accent color contrast warning.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Miyoung Choi miyoungc@nvidia.com

Summary by CodeRabbit

  • Documentation
    • Updated network policy guides with a safer export → validate → edit → reapply workflow for live policies.
    • Clarified starting from the sandbox’s current live policy and preserving onboarding presets.
    • Standardized OpenShell command examples, including openshell policy set --policy … --wait <sandbox-name> usage and raw-policy vs preset-style constraints.
  • UI/CLI Improvements
    • Updated the suggested openshell policy set command to include --wait <sandbox-name>.
  • Tests
    • Added coverage to ensure docs contain the correct command snippets and extraction/apply blocks.

Update raw policy replacement examples to use current OpenShell syntax.
Document the round-trippable base policy export before policy replacement.

Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
@miyoungc miyoungc added area: docs Documentation, examples, guides, or docs build area: policy Network policy, egress rules, presets, or sandbox policy labels Jun 26, 2026
@miyoungc miyoungc self-assigned this Jun 26, 2026
@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5853a21b-5ff1-4846-a416-83b864653ef0

📥 Commits

Reviewing files that changed from the base of the PR and between fc79f5c and 02a8235.

📒 Files selected for processing (7)
  • docs/network-policy/customize-network-policy.mdx
  • docs/network-policy/integration-policy-examples.mdx
  • docs/reference/cli-selection-guide.mdx
  • docs/reference/network-policies.mdx
  • src/lib/cli/public-dispatch.ts
  • test/cli/dispatch-basics.test.ts
  • test/policy-roundtrip-docs.test.ts
✅ Files skipped from review due to trivial changes (2)
  • src/lib/cli/public-dispatch.ts
  • docs/reference/network-policies.mdx
🚧 Files skipped from review as they are similar to previous changes (3)
  • docs/network-policy/integration-policy-examples.mdx
  • docs/reference/cli-selection-guide.mdx
  • docs/network-policy/customize-network-policy.mdx

📝 Walkthrough

Walkthrough

The network policy docs now show exporting the current live policy, extracting raw YAML into current-policy.yaml, validating required keys, and applying it back with openshell policy set --policy ... --wait. The CLI hint and tests were updated to match the new command form.

Changes

OpenShell network policy docs and CLI hints

Layer / File(s) Summary
Policy export and edit flow
docs/network-policy/customize-network-policy.mdx, docs/network-policy/integration-policy-examples.mdx, docs/reference/network-policies.mdx, docs/reference/cli-selection-guide.mdx
The policy workflow now exports the live policy with openshell policy get --full, strips the OpenShell header or YAML prefix, validates version and network_policies, writes current-policy.yaml, and reapplies it with openshell policy set --policy ... --wait. The policy preset note and network-policy guidance now describe the raw-policy-only set form.
Policy set hint and test
src/lib/cli/public-dispatch.ts, test/cli/dispatch-basics.test.ts
The OpenShell command suggestion for policy set now includes --wait <sandbox-name>, and the dispatch test expects that updated hint text.
Policy roundtrip docs tests
test/policy-roundtrip-docs.test.ts
A new Vitest suite checks the documentation for the updated get --full and set --policy ... --wait snippets, validates the extraction blocks, and ensures extraction and apply snippets are separate where expected.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

area: cli

Suggested reviewers

  • cv

Poem

A rabbit read the docs by moonlight, 🌙
Then twitched its nose and set things right.
awk did the trimming, --wait held steady,
current-policy.yaml made the hop quite ready.
🐇✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: fixing OpenShell policy round-trip examples in docs.
Linked Issues check ✅ Passed The docs now use the corrected get/set flow, strip metadata before reuse, and validate the exported YAML, addressing #5764.
Out of Scope Changes check ✅ Passed The added docs and test updates all support the policy round-trip fix and do not introduce unrelated scope.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/5764-policy-doc-roundtrip

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: docs-validation-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No merge-blocking E2E is required. The PR changes documentation, a CLI unknown-command hint string, and tests for those strings/snippets. It does not modify installer/onboarding state machines, sandbox lifecycle, credentials, security boundaries, network policy assets/enforcement, inference routing, deployment, or real assistant user-flow execution paths.

Optional E2E

  • docs-validation-e2e (low): Optional only: this PR changes several docs pages with command snippets. The nightly docs validation E2E can provide extra confidence that documentation examples and links remain healthy, but the PR already adds targeted non-live regression coverage and does not affect live runtime behavior.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: None
Optional Vitest E2E scenarios: None

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

  • None. No Vitest E2E scenario dispatch is required. The PR updates documentation, unit/docs tests outside test/e2e-scenario, and a CLI unknown-command OpenShell hint string; it does not change the Vitest scenario workflow, registry, live runtime support, fixtures, scenario metadata, onboarding/resume state-machine behavior, or a live-supported scenario surface.

Optional Vitest E2E scenarios

  • None.

Relevant changed files

  • src/lib/cli/public-dispatch.ts

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 2 test follow-ups
Since last review: 7 prior items resolved · 0 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add or identify a targeted OpenShell 0.0.44 doc-validation check that runs the documented `policy get --full <sandbox>` extraction pipeline and verifies `openshell policy set --policy current-policy.yaml --wait <sandbox>` is accepted.. Static tests cover the changed docs and CLI hint well, but this surface documents live sandbox policy replacement with the pinned OpenShell CLI. A runtime parser/doc-validation check would provide stronger confidence that OpenShell 0.0.44 accepts the exact documented argv shape and that representative `policy get --full` output round-trips as expected.
  • PRA-T2 Runtime validation — Add or identify a fixture-based shell-pipeline test for `policy get --full` output with metadata/status lines before `---`, confirming a valid `current-policy.yaml` is produced only for clean separator output and no editable policy file is produced for malformed or missing-separator output.. Static tests cover the changed docs and CLI hint well, but this surface documents live sandbox policy replacement with the pinned OpenShell CLI. A runtime parser/doc-validation check would provide stronger confidence that OpenShell 0.0.44 accepts the exact documented argv shape and that representative `policy get --full` output round-trips as expected.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/network-policy/customize-network-policy.mdx`:
- Around line 161-163: The `policy set` example uses the wrong argument order
and should match the command shape emitted by `buildPolicySetCommand()`. Update
the `openshell policy set` examples in this doc so the `--policy` flag comes
before the sandbox name and `--wait` stays in the same position, using the same
canonical order at both referenced occurrences.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cd4b8169-679e-43f2-aeb6-82ecdb453105

📥 Commits

Reviewing files that changed from the base of the PR and between 9f4d6ed and 2ad810f.

📒 Files selected for processing (4)
  • docs/network-policy/customize-network-policy.mdx
  • docs/network-policy/integration-policy-examples.mdx
  • docs/reference/cli-selection-guide.mdx
  • docs/reference/network-policies.mdx

Comment thread docs/network-policy/customize-network-policy.mdx
@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Changes requested

Merge posture: Do not merge yet
Primary next action: Resolve or justify PRA-1: Source-of-truth review needed: Awk pipeline workaround for OpenShell policy get --full metadata header.
Open items: 0 required · 2 warnings · 1 suggestion · 5 test follow-ups
Since last review: 6 prior items resolved · 1 still applies · 1 new item found

Action checklist

  • PRA-1 Resolve or justify: Source-of-truth review needed: Awk pipeline workaround for OpenShell policy get --full metadata header
  • PRA-2 Resolve or justify: Claimed regression test does not execute awk pipeline against live OpenShell output in test/policy-roundtrip-docs.test.ts:1
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Claimed regression test does not execute awk pipeline against live OpenShell output
  • PRA-T5 Add or justify test follow-up: Awk pipeline workaround for OpenShell policy get --full metadata header
  • PRA-3 In-scope improvement: Awk pipeline duplicated across 4 documentation files in docs/network-policy/customize-network-policy.mdx:158

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify tests test/policy-roundtrip-docs.test.ts:1 Either: (a) extend the test to run against a real OpenShell gateway in a test sandbox (requires test infrastructure), or (b) update the source-of-truth review comments to accurately reflect that the test validates documentation consistency only, not runtime pipeline correctness. Option (a) is preferred for true regression coverage.
PRA-3 Improvement scope docs/network-policy/customize-network-policy.mdx:158 Consider extracting the pipeline to a shared reference (e.g., a docs snippet include or a small shell script in the repo) so updates only need to happen in one place. Alternatively, add a code comment explaining why duplication is preferred over abstraction here.
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 1 in-scope improvement

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Awk pipeline workaround for OpenShell policy get --full metadata header

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: CLAIMED: test/policy-roundtrip-docs.test.ts validates this shared docs pattern. ACTUAL: Test only validates documentation string patterns. Does NOT execute pipeline against live openshell policy get --full output. Gap confirmed.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: Source-review comment blocks in all 4 .mdx files document all 5 fields. Test validates static patterns only.

PRA-2 Resolve/justify — Claimed regression test does not execute awk pipeline against live OpenShell output

  • Location: test/policy-roundtrip-docs.test.ts:1
  • Category: tests
  • Problem: The source-of-truth review comments in all 4 documentation files state 'regressionTest: test/policy-roundtrip-docs.test.ts validates this shared docs pattern', but the test only performs static text validation of documentation patterns (string matching on .mdx files). It does not execute the awk/grep pipeline against actual 'openshell policy get --full' output from a live sandbox, nor does it verify 'policy set --wait' accepts the result. This creates a false sense of security — the documented regression test doesn't actually test the runtime behavior that the workaround addresses.
  • Impact: If OpenShell changes its output format (e.g., stops emitting '---', adds new metadata headers, changes field names), the awk pipeline may silently fail or strip wrong content, leading to policy corruption on 'policy set'. The current test would not catch this regression.
  • Recommended action: Either: (a) extend the test to run against a real OpenShell gateway in a test sandbox (requires test infrastructure), or (b) update the source-of-truth review comments to accurately reflect that the test validates documentation consistency only, not runtime pipeline correctness. Option (a) is preferred for true regression coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run the test and observe it only reads .mdx files and checks string patterns — no openshell CLI invocation occurs. Check test/policy-roundtrip-docs.test.ts lines 30-75.
  • Missing regression test: Integration test that: (1) starts a test sandbox, (2) runs 'openshell policy get --full <sandbox>', (3) pipes through the documented awk/grep pipeline, (4) validates output is valid YAML with 'version:' and 'network_policies:' fields, (5) runs 'openshell policy set --policy <file> --wait <sandbox>' and confirms success.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run the test and observe it only reads .mdx files and checks string patterns — no openshell CLI invocation occurs. Check test/policy-roundtrip-docs.test.ts lines 30-75.
  • Evidence: Source-review comments in all 4 .mdx files claim this test as regressionTest; test/policy-roundtrip-docs.test.ts only reads files and asserts string patterns.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-3 Improvement — Awk pipeline duplicated across 4 documentation files

  • Location: docs/network-policy/customize-network-policy.mdx:158
  • Category: scope
  • Problem: The awk pipeline 'found { print } /^---$/ { found = 1 } END { if (!found) exit 1 }' is duplicated across 4 files (customize-network-policy.mdx, integration-policy-examples.mdx, cli-selection-guide.mdx, network-policies.mdx). While the new test validates consistency, the duplication creates maintenance risk — a future change to the pipeline would need to be applied in 4 places.
  • Impact: Maintenance burden and risk of inconsistency if pipeline needs updating (e.g., if OpenShell output format evolves).
  • Suggested action: Consider extracting the pipeline to a shared reference (e.g., a docs snippet include or a small shell script in the repo) so updates only need to happen in one place. Alternatively, add a code comment explaining why duplication is preferred over abstraction here.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: grep -n "found { print }" docs/network-policy/customize-network-policy.mdx docs/network-policy/integration-policy-examples.mdx docs/reference/cli-selection-guide.mdx docs/reference/network-policies.mdx
  • Missing regression test: Cross-file consistency check already exists in test/policy-roundtrip-docs.test.ts (validates identical awk pattern in all 4 files).
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Identical awk pipeline appears in all 4 documentation files at the metadata-stripping code blocks.
Simplification opportunities: 1 possible cut, net -40 lines possible

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-3 yagni (docs/network-policy/customize-network-policy.mdx:158): Duplicate awk pipeline in 4 documentation files
    • Replacement: Shared docs snippet or reference script
    • Net: -40 lines
    • Safety boundary: Must preserve fail-closed behavior (exit 1 if '---' header missing) and grep validations for 'version:' and 'network_policies:'
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — policy round-trip e2e: awk pipeline executes against real openshell policy get --full output — runs pipeline against live sandbox, validates YAML structure, confirms policy set --wait accepts it. Documentation examples describe a user-executed pipeline that processes live OpenShell output. The new test validates cross-file documentation consistency but does not execute the pipeline against real OpenShell output, leaving a runtime validation gap for the documented workaround.
  • PRA-T2 Runtime validation — policy round-trip negative: awk pipeline fails closed when --- header missing — feeds malformed output, expects exit 1. Documentation examples describe a user-executed pipeline that processes live OpenShell output. The new test validates cross-file documentation consistency but does not execute the pipeline against real OpenShell output, leaving a runtime validation gap for the documented workaround.
  • PRA-T3 Runtime validation — policy round-trip negative: grep validation rejects output missing version: or network_policies: — feeds incomplete YAML, expects validation failure. Documentation examples describe a user-executed pipeline that processes live OpenShell output. The new test validates cross-file documentation consistency but does not execute the pipeline against real OpenShell output, leaving a runtime validation gap for the documented workaround.
  • PRA-T4 Claimed regression test does not execute awk pipeline against live OpenShell output — Either: (a) extend the test to run against a real OpenShell gateway in a test sandbox (requires test infrastructure), or (b) update the source-of-truth review comments to accurately reflect that the test validates documentation consistency only, not runtime pipeline correctness. Option (a) is preferred for true regression coverage.
  • PRA-T5 Awk pipeline workaround for OpenShell policy get --full metadata header — CLAIMED: test/policy-roundtrip-docs.test.ts validates this shared docs pattern. ACTUAL: Test only validates documentation string patterns. Does NOT execute pipeline against live openshell policy get --full output. Gap confirmed.. Source-review comment blocks in all 4 .mdx files document all 5 fields. Test validates static patterns only.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Awk pipeline workaround for OpenShell policy get --full metadata header

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: CLAIMED: test/policy-roundtrip-docs.test.ts validates this shared docs pattern. ACTUAL: Test only validates documentation string patterns. Does NOT execute pipeline against live openshell policy get --full output. Gap confirmed.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: Source-review comment blocks in all 4 .mdx files document all 5 fields. Test validates static patterns only.

PRA-2 Resolve/justify — Claimed regression test does not execute awk pipeline against live OpenShell output

  • Location: test/policy-roundtrip-docs.test.ts:1
  • Category: tests
  • Problem: The source-of-truth review comments in all 4 documentation files state 'regressionTest: test/policy-roundtrip-docs.test.ts validates this shared docs pattern', but the test only performs static text validation of documentation patterns (string matching on .mdx files). It does not execute the awk/grep pipeline against actual 'openshell policy get --full' output from a live sandbox, nor does it verify 'policy set --wait' accepts the result. This creates a false sense of security — the documented regression test doesn't actually test the runtime behavior that the workaround addresses.
  • Impact: If OpenShell changes its output format (e.g., stops emitting '---', adds new metadata headers, changes field names), the awk pipeline may silently fail or strip wrong content, leading to policy corruption on 'policy set'. The current test would not catch this regression.
  • Recommended action: Either: (a) extend the test to run against a real OpenShell gateway in a test sandbox (requires test infrastructure), or (b) update the source-of-truth review comments to accurately reflect that the test validates documentation consistency only, not runtime pipeline correctness. Option (a) is preferred for true regression coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run the test and observe it only reads .mdx files and checks string patterns — no openshell CLI invocation occurs. Check test/policy-roundtrip-docs.test.ts lines 30-75.
  • Missing regression test: Integration test that: (1) starts a test sandbox, (2) runs 'openshell policy get --full <sandbox>', (3) pipes through the documented awk/grep pipeline, (4) validates output is valid YAML with 'version:' and 'network_policies:' fields, (5) runs 'openshell policy set --policy <file> --wait <sandbox>' and confirms success.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run the test and observe it only reads .mdx files and checks string patterns — no openshell CLI invocation occurs. Check test/policy-roundtrip-docs.test.ts lines 30-75.
  • Evidence: Source-review comments in all 4 .mdx files claim this test as regressionTest; test/policy-roundtrip-docs.test.ts only reads files and asserts string patterns.

PRA-3 Improvement — Awk pipeline duplicated across 4 documentation files

  • Location: docs/network-policy/customize-network-policy.mdx:158
  • Category: scope
  • Problem: The awk pipeline 'found { print } /^---$/ { found = 1 } END { if (!found) exit 1 }' is duplicated across 4 files (customize-network-policy.mdx, integration-policy-examples.mdx, cli-selection-guide.mdx, network-policies.mdx). While the new test validates consistency, the duplication creates maintenance risk — a future change to the pipeline would need to be applied in 4 places.
  • Impact: Maintenance burden and risk of inconsistency if pipeline needs updating (e.g., if OpenShell output format evolves).
  • Suggested action: Consider extracting the pipeline to a shared reference (e.g., a docs snippet include or a small shell script in the repo) so updates only need to happen in one place. Alternatively, add a code comment explaining why duplication is preferred over abstraction here.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: grep -n "found { print }" docs/network-policy/customize-network-policy.mdx docs/network-policy/integration-policy-examples.mdx docs/reference/cli-selection-guide.mdx docs/reference/network-policies.mdx
  • Missing regression test: Cross-file consistency check already exists in test/policy-roundtrip-docs.test.ts (validates identical awk pattern in all 4 files).
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Identical awk pipeline appears in all 4 documentation files at the metadata-stripping code blocks.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@miyoungc
miyoungc requested a review from cv June 26, 2026 19:57
Use the OpenShell 0.0.44-compatible full policy export and strip metadata before editing.
Keep the corrected name-first policy set syntax in the examples.

Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/network-policy/integration-policy-examples.mdx (1)

377-378: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use the actual policy set argv order.

Line 378 still shows openshell policy set <sandbox-name> --policy <file> --wait, but src/lib/policy/index.ts builds openshell policy set --policy <file> --wait <sandbox-name>. This example no longer matches the CLI contract the repo emits.

Suggested fix
-openshell policy set my-assistant --policy current-policy.yaml --wait
+openshell policy set --policy current-policy.yaml --wait my-assistant
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/network-policy/integration-policy-examples.mdx` around lines 377 - 378,
The example for the policy set command uses the wrong argv order compared with
the CLI contract emitted by `src/lib/policy/index.ts`. Update the documented
`openshell policy set` example to match the actual order built by the policy
command path, keeping the same symbols (`policy set`, `--policy`, `--wait`, and
the sandbox name) but placing them in the order used by the implementation so
the docs and generated CLI behavior stay aligned.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/network-policy/integration-policy-examples.mdx`:
- Around line 377-378: The example for the policy set command uses the wrong
argv order compared with the CLI contract emitted by `src/lib/policy/index.ts`.
Update the documented `openshell policy set` example to match the actual order
built by the policy command path, keeping the same symbols (`policy set`,
`--policy`, `--wait`, and the sandbox name) but placing them in the order used
by the implementation so the docs and generated CLI behavior stay aligned.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7245fcac-2bb6-4d29-9682-e0175bc04d8e

📥 Commits

Reviewing files that changed from the base of the PR and between 2ad810f and b07efa7.

📒 Files selected for processing (4)
  • docs/network-policy/customize-network-policy.mdx
  • docs/network-policy/integration-policy-examples.mdx
  • docs/reference/cli-selection-guide.mdx
  • docs/reference/network-policies.mdx
✅ Files skipped from review due to trivial changes (1)
  • docs/reference/network-policies.mdx

@github-code-quality

github-code-quality Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the fix/5764-policy-doc-... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/5764-policy-doc-... 02a8235 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the fix/5764-policy-doc-... branch is 47%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/5764-policy-doc-... 02a8235 +/-
src/lib/state/o...oard-session.ts 91%
src/lib/actions...dbox/rebuild.ts 73%
src/lib/sandbox/config.ts 72%
src/lib/onboard/preflight.ts 62%
src/lib/shields/index.ts 62%
src/lib/actions...licy-channel.ts 60%
src/lib/state/sandbox.ts 56%
src/lib/policy/index.ts 48%
src/lib/onboard...er-gpu-patch.ts 47%
src/lib/onboard.ts 19%

Updated June 26, 2026 22:02 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Comment thread ci/platform-matrix.json
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
cv merged commit d0e175f into main Jun 26, 2026
47 checks passed
@cv
cv deleted the fix/5764-policy-doc-roundtrip branch June 26, 2026 22:13
cv pushed a commit that referenced this pull request Jun 29, 2026
## Summary
Adds the v0.0.69 release notes to the published release-notes page so
users can see the shipped sandbox recovery, Deep Agents Code, Hermes,
inference, policy, and release-validation changes.
The section is based on the v0.0.69 announcement and links each
user-facing theme to the deeper docs pages that already cover the
behavior.

## Changes
- Added a new `v0.0.69` section to `docs/about/release-notes.mdx`.
- Linked release-note themes to lifecycle, backup, troubleshooting, Deep
Agents Code, commands, workspace, messaging, Hermes, inference,
security, monitoring, and network-policy docs.

Source summary:
- #5455 -> `docs/about/release-notes.mdx`: Summarized persistent
workspace and state cleanup during sandbox destroy.
- #5738 -> `docs/about/release-notes.mdx`: Summarized nonzero exit
status preservation for failed hosted endpoint validation.
- #5786 -> `docs/about/release-notes.mdx`: Summarized live sandbox
rediscovery when local registry state is missing.
- #5881 -> `docs/about/release-notes.mdx`: Summarized the
`nemo-deepagents` alias command surface.
- #5594 -> `docs/about/release-notes.mdx`: Summarized the Hermes Agent
2026.6.19 update.
- #5777 -> `docs/about/release-notes.mdx`: Summarized manifest-derived
messaging channel support.
- #5825 -> `docs/about/release-notes.mdx`: Summarized DeepSeek V4 Flash
managed-vLLM defaults for DGX Station.
- #5877 -> `docs/about/release-notes.mdx`: Summarized provider switch
metadata preservation.
- #5932 -> `docs/about/release-notes.mdx`: Summarized transient
inference smoke retry behavior.
- #5934 -> `docs/about/release-notes.mdx`: Summarized constrained
inference smoke retry boundaries.
- #5681 -> `docs/about/release-notes.mdx`: Summarized Shields
config-hash sealing during auto-restore.
- #5682 -> `docs/about/release-notes.mdx`: Summarized sandbox connect
process-limit enforcement.
- #5683 -> `docs/about/release-notes.mdx`: Summarized JSON agent failure
provenance warnings.
- #5711 -> `docs/about/release-notes.mdx`: Summarized sparse-source log
breadcrumbs.
- #5838 -> `docs/about/release-notes.mdx`: Summarized host-authoritative
Shields status.
- #5880 -> `docs/about/release-notes.mdx`: Summarized policy round-trip
documentation updates.
- #5886 -> `docs/about/release-notes.mdx`: Summarized network request
approval-flow documentation updates.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: doc-only release-notes
prose; no runtime behavior changed.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [ ] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

`npm run docs` passed with 0 errors and the existing Fern light-mode
accent contrast warning.
`fern check --warnings` reported the same accent-color warning.

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added release notes for **v0.0.69**, covering improved sandbox
lifecycle recovery (state preservation across
destroy/recreate/rebuild/recovery/validation failures), clearer Deep
Agents Code terminal/CLI behavior, and safer Hermes messaging/provider
switching with manifest-driven channels.
* Improved inference setup validation guidance, including handling of
local/compatible endpoints and redaction of sensitive validation errors.
* Refreshed release-gate documentation with clearer approval examples
and validation behavior for NVIDIA API keys vs hosted inference keys.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
## Summary
Update the raw OpenShell policy replacement examples so they match
NemoClaw's pinned OpenShell 0.0.44 command surface.
The examples now export the full live policy with `openshell policy get
<name> --full`, strip the OpenShell metadata header before editing, and
apply the file back with the name-first `policy set` syntax.

## Related Issue
Fixes NVIDIA#5764

## Changes
- Update the customize network policy guide to use the pinned-compatible
`openshell policy get <name> --full` export before raw policy
replacement.
- Add a fail-closed metadata-header stripping step before writing
`current-policy.yaml` for manual edits.
- Align related policy snippets in integration examples, the CLI
selection guide, and the network policy reference with OpenShell 0.0.44
syntax.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [x] Doc only (includes code sample changes)

## Quality Gates
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: doc-only correction verified
against the OpenShell 0.0.44 CLI parser and published policy docs.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: docs-only policy
command examples checked against OpenShell 0.0.44
`PolicyCommands::Set/Get`; no runtime policy enforcement code changed.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [ ] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — `npm
run docs` passed, but Fern reports an existing light-mode accent color
contrast warning.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated network policy guides with a safer export → validate → edit →
reapply workflow for live policies.
* Clarified starting from the sandbox’s current live policy and
preserving onboarding presets.
* Standardized OpenShell command examples, including `openshell policy
set --policy … --wait <sandbox-name>` usage and raw-policy vs
preset-style constraints.
* **UI/CLI Improvements**
* Updated the suggested `openshell policy set` command to include
`--wait <sandbox-name>`.
* **Tests**
* Added coverage to ensure docs contain the correct command snippets and
extraction/apply blocks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
## Summary
Adds the v0.0.69 release notes to the published release-notes page so
users can see the shipped sandbox recovery, Deep Agents Code, Hermes,
inference, policy, and release-validation changes.
The section is based on the v0.0.69 announcement and links each
user-facing theme to the deeper docs pages that already cover the
behavior.

## Changes
- Added a new `v0.0.69` section to `docs/about/release-notes.mdx`.
- Linked release-note themes to lifecycle, backup, troubleshooting, Deep
Agents Code, commands, workspace, messaging, Hermes, inference,
security, monitoring, and network-policy docs.

Source summary:
- NVIDIA#5455 -> `docs/about/release-notes.mdx`: Summarized persistent
workspace and state cleanup during sandbox destroy.
- NVIDIA#5738 -> `docs/about/release-notes.mdx`: Summarized nonzero exit
status preservation for failed hosted endpoint validation.
- NVIDIA#5786 -> `docs/about/release-notes.mdx`: Summarized live sandbox
rediscovery when local registry state is missing.
- NVIDIA#5881 -> `docs/about/release-notes.mdx`: Summarized the
`nemo-deepagents` alias command surface.
- NVIDIA#5594 -> `docs/about/release-notes.mdx`: Summarized the Hermes Agent
2026.6.19 update.
- NVIDIA#5777 -> `docs/about/release-notes.mdx`: Summarized manifest-derived
messaging channel support.
- NVIDIA#5825 -> `docs/about/release-notes.mdx`: Summarized DeepSeek V4 Flash
managed-vLLM defaults for DGX Station.
- NVIDIA#5877 -> `docs/about/release-notes.mdx`: Summarized provider switch
metadata preservation.
- NVIDIA#5932 -> `docs/about/release-notes.mdx`: Summarized transient
inference smoke retry behavior.
- NVIDIA#5934 -> `docs/about/release-notes.mdx`: Summarized constrained
inference smoke retry boundaries.
- NVIDIA#5681 -> `docs/about/release-notes.mdx`: Summarized Shields
config-hash sealing during auto-restore.
- NVIDIA#5682 -> `docs/about/release-notes.mdx`: Summarized sandbox connect
process-limit enforcement.
- NVIDIA#5683 -> `docs/about/release-notes.mdx`: Summarized JSON agent failure
provenance warnings.
- NVIDIA#5711 -> `docs/about/release-notes.mdx`: Summarized sparse-source log
breadcrumbs.
- NVIDIA#5838 -> `docs/about/release-notes.mdx`: Summarized host-authoritative
Shields status.
- NVIDIA#5880 -> `docs/about/release-notes.mdx`: Summarized policy round-trip
documentation updates.
- NVIDIA#5886 -> `docs/about/release-notes.mdx`: Summarized network request
approval-flow documentation updates.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: doc-only release-notes
prose; no runtime behavior changed.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [ ] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

`npm run docs` passed with 0 errors and the existing Fern light-mode
accent contrast warning.
`fern check --warnings` reported the same accent-color warning.

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added release notes for **v0.0.69**, covering improved sandbox
lifecycle recovery (state preservation across
destroy/recreate/rebuild/recovery/validation failures), clearer Deep
Agents Code terminal/CLI behavior, and safer Hermes messaging/provider
switching with manifest-driven channels.
* Improved inference setup validation guidance, including handling of
local/compatible endpoints and redaction of sensitive validation errors.
* Refreshed release-gate documentation with clearer approval examples
and validation behavior for NVIDIA API keys vs hosted inference keys.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: docs Documentation, examples, guides, or docs build area: policy Network policy, egress rules, presets, or sandbox policy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[All Platforms][Docs] NemoClaw Customize Network Policy — openshell policy set command fails as documented

3 participants