Skip to content

chore: refresh and re-sign nemoclaw-user-reference skill #4782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
miyoungc merged 2 commits into from
Jun 4, 2026
Merged

chore: refresh and re-sign nemoclaw-user-reference skill #4782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7538a08
chore: refresh user reference skill
miyoungc Jun 4, 2026
41b670d
Attach NVSkills validation signatures
svc-nvskills-signing Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 44 additions & 24 deletions skills/nemoclaw-user-reference/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,19 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `nemoclaw-user-reference`
- Evaluation date: 2026-05-28
- Evaluation date: 2026-06-04
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 1 evaluation tasks
- Attempts per task: 2
- Pass threshold: 50%
- Overall verdict: FAIL
- Tier 3 live agent evaluation: not available in this report
The skill should be reviewed before NVSkills-Eval publication. **Skill owners should address the applicable findings below and rerun NVSkills-Eval to refresh this benchmark.**

## Agents Used

- Tier 3 agent details were not available in this report.
- `claude-code`
- `codex`

## Metrics Used

Expand All @@ -28,45 +33,60 @@ Reported benchmark dimensions:

Underlying evaluation signals used in this run:

- No Tier 3 evaluation signal details were available in this report.
- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.

## Test Tasks

Tier 3 evaluation task details were not available in this report.
The benchmark dataset contained 1 evaluation tasks:

- Positive tasks: 1 tasks where the skill was expected to activate.
- Negative tasks: 0 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.

## Results

Tier 3 dimension rollup was not available in this report.
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 100% (+62%) | 92% (+50%) |
| Discoverability | 2 | 100% (+38%) | 76% (+22%) |
| Effectiveness | 2 | 93% (+59%) | 91% (+54%) |
| Efficiency | 2 | 88% (+32%) | 67% (+24%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 14 total findings.
Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 13 total findings.

Top findings:

- MEDIUM PII/ip_addresses: Non-RFC1918 IP address (`references/troubleshooting.md:137`)
- MEDIUM QUALITY/quality_correctness: Guide-only skill has very little content (14 lines) (`skills/nemoclaw-user-reference/SKILL.md`)
- MEDIUM PII/ip_addresses: Non-RFC1918 IP address (`references/troubleshooting.md:135`)
- MEDIUM QUALITY/quality_correctness: Guide-only skill has very little content (12 lines) (`skills/nemoclaw-user-reference/SKILL.md`)
- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/nemoclaw-user-reference/SKILL.md`)
- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/nemoclaw-user-reference/SKILL.md`)
- MEDIUM QUALITY/quality_efficiency: Deeply nested references in troubleshooting.md (`skills/nemoclaw-user-reference/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemoclaw-user-reference/SKILL.md`)

## Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 2 total findings.
Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 3 total findings.

Top findings:

- HIGH DUPLICATE/duplicate: Duplicate content found across SKILL.md and references/architecture.md and references/cli-selection-guide.md and references/commands.md and references/network-policies.md and references/troubleshooting.md:
"(preamble)" in SKILL.md (lines 1-3)
vs "(preamble)" in references/architecture.md (lines 1-2)
vs "(preamble)" in references/cli-selection-guide.md (lines 1-2)
vs "(preamble)" in references/commands.md (lines 1-2)
vs "(preamble)" in references/network-policies.md (lines 1-2)
vs "(preamble)" in references/troubleshooting.md (lines 1-2) (`SKILL.md:1`)
- HIGH DUPLICATE/duplicate: Duplicate content found within references/commands.md:
"### Onboarding Configuration" in references/commands.md (lines 1235-1236)
vs "### Onboarding Behavior Flags" in references/commands.md (lines 1278-1278) (`references/commands.md:1235`)

## Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.
"#### `--from <Dockerfile>`" in references/commands.md (lines 298-334)
vs "### `nemoclaw onboard --from`" in references/commands.md (lines 350-358) (`references/commands.md:298`)
- HIGH DUPLICATE/duplicate: Duplicate content found within references/commands.md:
"#### `--resume` and `--fresh`" in references/commands.md (lines 124-127)
vs "#### `--resume` and `--fresh`" in references/commands.md (lines 131-131) (`references/commands.md:124`)
- HIGH DUPLICATE/duplicate: Duplicate content found within references/commands.md:
"#### `--resume` and `--fresh`" in references/commands.md (lines 208-211)
vs "### `nemoclaw <name> channels add <channel>`" in references/commands.md (lines 881-884) (`references/commands.md:208`)
13 changes: 5 additions & 8 deletions skills/nemoclaw-user-reference/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
---

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add required SPDX header at the top of this file.

SKILL.md is missing the required SPDX copyright/license header.

As per coding guidelines, all *.md files must include SPDX headers using HTML comments.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemoclaw-user-reference/SKILL.md` at line 1, Add the required SPDX
header as HTML comments at the very top of SKILL.md: insert an SPDX copyright
line (SPDX-FileCopyrightText) and an SPDX license identifier line
(SPDX-License-Identifier) wrapped in <!-- --> so the file begins with those two
HTML comment lines; ensure the copyright holder/year and the correct license
identifier are filled in.

name: "nemoclaw-user-reference"
description: "Describes the NemoClaw plugin and blueprint architecture and how they orchestrate the OpenClaw sandbox. Use when looking up architecture, plugin structure, or blueprint design. Trigger keywords - nemoclaw architecture, nemoclaw plugin blueprint structure, nemoclaw vs openshell, which cli, nemoclaw cli, openshell cli, sandbox commands, nemoclaw cli commands, nemoclaw command reference, nemoclaw network policy, sandbox egress control operator approval, nemoclaw troubleshooting, nemoclaw debug sandbox issues."
description: "Describes the NemoClaw integration layer and blueprint architecture and how they orchestrate compatible agent sandboxes. Use when looking up architecture, agent integration, plugin structure, or blueprint design. Trigger keywords - nemoclaw architecture, nemoclaw agent architecture, nemoclaw plugin blueprint structure, nemoclaw vs openshell, which cli, nemoclaw cli, openshell cli, sandbox commands, nemoclaw cli commands, nemoclaw command reference, nemoclaw network policy, sandbox egress control operator approval, nemoclaw troubleshooting, nemoclaw debug sandbox issues."
license: "Apache-2.0"
---

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Architecture Details
# NemoClaw User Reference

## References

- **Load [references/architecture.md](references/architecture.md)** when looking up architecture, plugin structure, or blueprint design. Describes the NemoClaw plugin and blueprint architecture and how they orchestrate the OpenClaw sandbox.
- **[references/cli-selection-guide.md](references/cli-selection-guide.md)** — Explains when to use `nemoclaw` versus `openshell` for NemoClaw-managed sandboxes, including lifecycle, inference, policy, monitoring, file transfer, and gateway operations.
- **Load [references/commands.md](references/commands.md)** when looking up a specific `nemoclaw` or `/nemoclaw` subcommand, flag, argument, or exit code. Includes the full CLI reference for slash commands and standalone NemoClaw commands.
- **Load [references/architecture.md](references/architecture.md)** when looking up architecture, agent integration, plugin structure, or blueprint design. Describes the NemoClaw integration layer and blueprint architecture and how they orchestrate compatible agent sandboxes.
- **[references/cli-selection-guide.md](references/cli-selection-guide.md)** — Explains when to use `$$nemoclaw` versus `openshell` for NemoClaw-managed sandboxes, including lifecycle, inference, policy, monitoring, file transfer, and gateway operations.
- **Load [references/commands.md](references/commands.md)** when looking up a specific `$$nemoclaw`, `nemohermes`, or `/nemoclaw` subcommand, flag, argument, or exit code. Includes the full CLI reference for standalone NemoClaw commands and agent-specific in-sandbox commands.
- **Load [references/network-policies.md](references/network-policies.md)** when looking up a specific default endpoint, filesystem path, or the runtime approval sequence NemoClaw applies on blocked requests. Covers the baseline network policy, filesystem rules, and operator approval flow.
- **Load [references/troubleshooting.md](references/troubleshooting.md)** when diagnosing a reported NemoClaw error, a failed onboard, or unexpected sandbox behavior. Lists fixes for common installation, onboarding, and runtime issues.
89 changes: 4 additions & 85 deletions skills/nemoclaw-user-reference/evals/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,90 +3,9 @@
"id": "docs-reference-architecture-001",
"question": "I'm using the architecture reference. Help me verify implementation and operations details so I can make changes or debug behavior from the right mental model.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user verify implementation and operations details and gives enough concrete guidance, decision criteria, verification steps, or risk framing to make changes or debug behavior from the right mental model."
},
{
"id": "docs-reference-architecture-002",
"question": "I'm locating the right layer for a change. Help me connect structure, lifecycle, and ownership so I can edit the plugin, blueprint, or sandbox layer intentionally.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user connect structure, lifecycle, and ownership and gives enough concrete guidance, decision criteria, verification steps, or risk framing to edit the plugin, blueprint, or sandbox layer intentionally."
},
{
"id": "docs-reference-architecture-003",
"question": "I'm modifying architecture-sensitive files. Help me understand the invariants that must remain true so I can avoid breaking sandbox creation or operation.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user understand the invariants that must remain true and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid breaking sandbox creation or operation."
},
{
"id": "docs-reference-cli-selection-guide-001",
"question": "I'm choosing between `nemoclaw` and `openshell`. Help me pick the command surface that owns my operation so I can complete the task without breaking NemoClaw management.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user pick the command surface that owns my operation and gives enough concrete guidance, decision criteria, verification steps, or risk framing to complete the task without breaking NemoClaw management."
},
{
"id": "docs-reference-cli-selection-guide-002",
"question": "I'm tempted to run a low-level OpenShell command. Help me understand when it bypasses NemoClaw lifecycle expectations so I can avoid unsupported or confusing states.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user understand when it bypasses NemoClaw lifecycle expectations and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid unsupported or confusing states."
},
{
"id": "docs-reference-cli-selection-guide-003",
"question": "I'm doing onboarding, policy, debugging, or low-level sandbox work. Help me apply a simple CLI selection rule so I can move quickly without memorizing every command boundary.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user apply a simple CLI selection rule and gives enough concrete guidance, decision criteria, verification steps, or risk framing to move quickly without memorizing every command boundary."
},
{
"id": "docs-reference-commands-001",
"question": "I'm opening the CLI command reference. Help me find the exact action, flag, or recovery path I need so I can run the right command without scanning source code.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user find the exact action, flag, or recovery path I need and gives enough concrete guidance, decision criteria, verification steps, or risk framing to run the right command without scanning source code."
},
{
"id": "docs-reference-commands-002",
"question": "I'm choosing among setup, lifecycle, policy, inference, and diagnostic commands. Help me see command groups and examples clearly so I can locate the right command faster.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user see command groups and examples clearly and gives enough concrete guidance, decision criteria, verification steps, or risk framing to locate the right command faster."
},
{
"id": "docs-reference-commands-003",
"question": "I'm using `nemoclaw` in scripts or CI. Help me understand example output and exit behavior so I can automate safely and handle failures correctly.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user understand example output and exit behavior and gives enough concrete guidance, decision criteria, verification steps, or risk framing to automate safely and handle failures correctly."
},
{
"id": "docs-reference-network-policies-001",
"question": "I'm reading the network policy reference. Help me understand baseline access decisions so I can assess the sandbox's default exposure.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user understand baseline access decisions and gives enough concrete guidance, decision criteria, verification steps, or risk framing to assess the sandbox's default exposure."
},
{
"id": "docs-reference-network-policies-002",
"question": "I'm reviewing default rules and filesystem constraints. Help me know what is allowed before any customization so I can decide what needs to change for my environment.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user know what is allowed before any customization and gives enough concrete guidance, decision criteria, verification steps, or risk framing to decide what needs to change for my environment."
},
{
"id": "docs-reference-network-policies-003",
"question": "I'm tracing a denied request. Help me understand how operator approval turns into policy so I can audit and reason about egress changes.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user understand how operator approval turns into policy and gives enough concrete guidance, decision criteria, verification steps, or risk framing to audit and reason about egress changes."
},
{
"id": "docs-reference-troubleshooting-001",
"question": "I'm troubleshooting a NemoClaw problem. Help me map my symptom to the fastest safe recovery step so I can get unstuck without making the environment worse.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user map my symptom to the fastest safe recovery step and gives enough concrete guidance, decision criteria, verification steps, or risk framing to get unstuck without making the environment worse."
},
{
"id": "docs-reference-troubleshooting-002",
"question": "I'm uncertain which subsystem failed. Help me run diagnostics that separate install, onboarding, runtime, policy, and inference issues so I can focus on the likely cause.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user run diagnostics that separate install, onboarding, runtime, policy, and inference issues and gives enough concrete guidance, decision criteria, verification steps, or risk framing to focus on the likely cause."
},
{
"id": "docs-reference-troubleshooting-003",
"question": "I'm asking the community or filing an issue. Help me collect the right logs, versions, and error details so I can give maintainers enough information to help.",
"expected_skill": "nemoclaw-user-reference",
"ground_truth": "A NemoClaw-specific answer that helps the user collect the right logs, versions, and error details and gives enough concrete guidance, decision criteria, verification steps, or risk framing to give maintainers enough information to help."
"ground_truth": "A NemoClaw-specific answer that helps the user verify implementation and operations details and gives enough concrete guidance, decision criteria, verification steps, or risk framing to make changes or debug behavior from the right mental model.",
"expected_behavior": [
"Uses the expected_skill and does not make up answers if it cannot find the answer from the skill."
]
}
]
Loading
Loading