Skip to content

chore: skills signing batch 2 #4463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
miyoungc merged 3 commits into from
May 28, 2026
Merged

chore: skills signing batch 2 #4463

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aafdce3
chore: 2nd skills signing batch
miyoungc May 28, 2026
dc1bfc7
Attach NVSkills validation signatures
svc-nvskills-signing May 28, 2026
4b55ec7
Merge branch 'main' into skills-sign-batch-2
jyaunches May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 15 additions & 135 deletions .agents/skills/nemoclaw-user-configure-inference/evals/evals.json
View file
Open in desktop

Large diffs are not rendered by default.

90 changes: 9 additions & 81 deletions .agents/skills/nemoclaw-user-configure-security/evals/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,126 +3,54 @@
"id": "docs-security-best-practices-001",
"question": "I'm evaluating NemoClaw security best practices. Help me understand the risk posture of each configurable control so I can justify the setup to my team or security reviewers.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user understand the risk posture of each configurable control and gives enough concrete guidance, decision criteria, verification steps, or risk framing to justify the setup to my team or security reviewers.",
"expected_behavior": [
"The output directly addresses the user's situation: evaluating NemoClaw security best practices.",
"The AI coding assistant loads the expected_skill and references/best-practices.md",
"The output helps the user understand the risk posture of each configurable control with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to justify the setup to my team or security reviewers.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/best-practices.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user understand the risk posture of each configurable control and gives enough concrete guidance, decision criteria, verification steps, or risk framing to justify the setup to my team or security reviewers."
},
{
"id": "docs-security-best-practices-002",
"question": "I'm balancing developer convenience with lockdown. Help me compare the trade-offs of changing security controls so I can choose a posture that fits the environment.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user compare the trade-offs of changing security controls and gives enough concrete guidance, decision criteria, verification steps, or risk framing to choose a posture that fits the environment.",
"expected_behavior": [
"The output directly addresses the user's situation: balancing developer convenience with lockdown.",
"The AI coding assistant loads the expected_skill and references/best-practices.md",
"The output helps the user compare the trade-offs of changing security controls with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to choose a posture that fits the environment.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/best-practices.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user compare the trade-offs of changing security controls and gives enough concrete guidance, decision criteria, verification steps, or risk framing to choose a posture that fits the environment."
},
{
"id": "docs-security-best-practices-003",
"question": "I'm preparing for production-like use. Help me see which defaults are acceptable and which require changes so I can avoid shipping with accidental weak spots.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user see which defaults are acceptable and which require changes and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid shipping with accidental weak spots.",
"expected_behavior": [
"The output directly addresses the user's situation: preparing for production-like use.",
"The AI coding assistant loads the expected_skill and references/best-practices.md",
"The output helps the user see which defaults are acceptable and which require changes with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to avoid shipping with accidental weak spots.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/best-practices.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user see which defaults are acceptable and which require changes and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid shipping with accidental weak spots."
},
{
"id": "docs-security-credential-storage-001",
"question": "I'm inspecting NemoClaw credential storage. Help me verify how secrets are stored and protected so I can decide whether the setup meets my secret-handling expectations.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user verify how secrets are stored and protected and gives enough concrete guidance, decision criteria, verification steps, or risk framing to decide whether the setup meets my secret-handling expectations.",
"expected_behavior": [
"The output directly addresses the user's situation: inspecting NemoClaw credential storage.",
"The AI coding assistant loads the expected_skill and references/credential-storage.md",
"The output helps the user verify how secrets are stored and protected with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to decide whether the setup meets my secret-handling expectations.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/credential-storage.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user verify how secrets are stored and protected and gives enough concrete guidance, decision criteria, verification steps, or risk framing to decide whether the setup meets my secret-handling expectations."
},
{
"id": "docs-security-credential-storage-002",
"question": "I'm tracing where credentials live. Help me distinguish host, gateway, and sandbox storage boundaries so I can avoid assuming secrets are available in the wrong place.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user distinguish host, gateway, and sandbox storage boundaries and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid assuming secrets are available in the wrong place.",
"expected_behavior": [
"The output directly addresses the user's situation: tracing where credentials live.",
"The AI coding assistant loads the expected_skill and references/credential-storage.md",
"The output helps the user distinguish host, gateway, and sandbox storage boundaries with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to avoid assuming secrets are available in the wrong place.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/credential-storage.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user distinguish host, gateway, and sandbox storage boundaries and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid assuming secrets are available in the wrong place."
},
{
"id": "docs-security-credential-storage-003",
"question": "I'm rotating or inspecting credentials. Help me follow a workflow that does not print secrets in logs or docs so I can recover or update access safely.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user follow a workflow that does not print secrets in logs or docs and gives enough concrete guidance, decision criteria, verification steps, or risk framing to recover or update access safely.",
"expected_behavior": [
"The output directly addresses the user's situation: rotating or inspecting credentials.",
"The AI coding assistant loads the expected_skill and references/credential-storage.md",
"The output helps the user follow a workflow that does not print secrets in logs or docs with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to recover or update access safely.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/credential-storage.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user follow a workflow that does not print secrets in logs or docs and gives enough concrete guidance, decision criteria, verification steps, or risk framing to recover or update access safely."
},
{
"id": "docs-security-openclaw-controls-001",
"question": "I'm reading about controls outside NemoClaw's scope. Help me understand which security responsibilities remain with OpenClaw so I can avoid treating sandbox isolation as a complete application security model.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user understand which security responsibilities remain with OpenClaw and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid treating sandbox isolation as a complete application security model.",
"expected_behavior": [
"The output directly addresses the user's situation: reading about controls outside NemoClaw's scope.",
"The AI coding assistant loads the expected_skill and references/openclaw-controls.md",
"The output helps the user understand which security responsibilities remain with OpenClaw with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to avoid treating sandbox isolation as a complete application security model.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/openclaw-controls.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user understand which security responsibilities remain with OpenClaw and gives enough concrete guidance, decision criteria, verification steps, or risk framing to avoid treating sandbox isolation as a complete application security model."
},
{
"id": "docs-security-openclaw-controls-002",
"question": "I'm assessing application-layer agent risk. Help me identify the controls NemoClaw does not add so I can plan separate mitigations for authentication, prompt handling, and agent behavior.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user identify the controls NemoClaw does not add and gives enough concrete guidance, decision criteria, verification steps, or risk framing to plan separate mitigations for authentication, prompt handling, and agent behavior.",
"expected_behavior": [
"The output directly addresses the user's situation: assessing application-layer agent risk.",
"The AI coding assistant loads the expected_skill and references/openclaw-controls.md",
"The output helps the user identify the controls NemoClaw does not add with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to plan separate mitigations for authentication, prompt handling, and agent behavior.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/openclaw-controls.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user identify the controls NemoClaw does not add and gives enough concrete guidance, decision criteria, verification steps, or risk framing to plan separate mitigations for authentication, prompt handling, and agent behavior."
},
{
"id": "docs-security-openclaw-controls-003",
"question": "I'm documenting the security boundary. Help me explain where NemoClaw protection ends so I can set accurate expectations for reviewers and operators.",
"expected_skill": "nemoclaw-user-configure-security",
"ground_truth": "A NemoClaw-specific answer that helps the user explain where NemoClaw protection ends and gives enough concrete guidance, decision criteria, verification steps, or risk framing to set accurate expectations for reviewers and operators.",
"expected_behavior": [
"The output directly addresses the user's situation: documenting the security boundary.",
"The AI coding assistant loads the expected_skill and references/openclaw-controls.md",
"The output helps the user explain where NemoClaw protection ends with NemoClaw-specific guidance rather than generic advice.",
"The output gives enough concrete guidance, decision criteria, verification steps, or risk framing for the user to set accurate expectations for reviewers and operators.",
"The output avoids inventing unsupported NemoClaw behavior.",
"The output follows progressive disclosure: it answers the current request without dumping unrelated details other than the expected_skill and the references/openclaw-controls.md file."
]
"ground_truth": "A NemoClaw-specific answer that helps the user explain where NemoClaw protection ends and gives enough concrete guidance, decision criteria, verification steps, or risk framing to set accurate expectations for reviewers and operators."
}
]
75 changes: 75 additions & 0 deletions skills/nemoclaw-user-configure-inference/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Evaluation Report

Evaluation of the `nemoclaw-user-configure-inference` skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `nemoclaw-user-configure-inference`
- Evaluation date: 2026-05-28
- NVSkills-Eval profile: `external`
- Overall verdict: FAIL
- Tier 3 live agent evaluation: not available in this report

## Agents Used

- Tier 3 agent details were not available in this report.

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

Tier 3 evaluation task details were not available in this report.

## Results

Tier 3 dimension rollup was not available in this report.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 13 total findings.

Top findings:

- MEDIUM PII/gps_coordinates: GPS coordinates (location information) (`references/inference-options.md:89`)
- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/nemoclaw-user-configure-inference/SKILL.md`)
- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/nemoclaw-user-configure-inference/SKILL.md`)
- MEDIUM QUALITY/quality_efficiency: Deeply nested references in set-up-sub-agent.md (`skills/nemoclaw-user-configure-inference/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemoclaw-user-configure-inference/SKILL.md`)

## Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 3 total findings.

Top findings:

- HIGH DUPLICATE/duplicate: Duplicate content found across SKILL.md and references/inference-options.md and references/set-up-sub-agent.md and references/switch-inference-providers.md and references/tool-calling-reliability.md and references/use-local-inference-details.md:
"(preamble)" in SKILL.md (lines 1-3)
vs "(preamble)" in references/inference-options.md (lines 1-2)
vs "(preamble)" in references/set-up-sub-agent.md (lines 1-2)
vs "(preamble)" in references/switch-inference-providers.md (lines 1-2)
vs "(preamble)" in references/tool-calling-reliability.md (lines 1-2)
vs "(preamble)" in references/use-local-inference-details.md (lines 1-2) (`SKILL.md:1`)
- HIGH DUPLICATE/duplicate: Duplicate content found across references/inference-options.md and references/tool-calling-reliability.md:
"## Next Steps" in references/inference-options.md (lines 138-142)
vs "## Next Steps" in references/tool-calling-reliability.md (lines 160-164) (`references/inference-options.md:138`)
- HIGH DUPLICATE/duplicate: Duplicate content found across references/inference-options.md and references/switch-inference-providers.md:
"## How Inference Routing Works" in references/inference-options.md (lines 9-19)
vs "## Notes" in references/switch-inference-providers.md (lines 197-204) (`references/inference-options.md:9`)

## Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.
Loading