Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .agents/rules/cuopt-skill-evolution.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
description: Trigger skill evolution — after solving non-trivial problems, propose skill updates.
alwaysApply: true
---

# Skill Evolution

After resolving a non-trivial problem, read and follow `skills/cuopt-skill-evolution/SKILL.md` to check whether the learning should be captured as a skill update.
8 changes: 0 additions & 8 deletions .agents/rules/skill-evolution.mdc

This file was deleted.

4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@
"description": "cuOpt REST server — start server, endpoints, Python/curl client examples. Use when the user is deploying or calling the REST API."
},
{
"name": "skill-evolution",
"source": "./skills/skill-evolution",
"name": "cuopt-skill-evolution",
"source": "./skills/cuopt-skill-evolution",
"skills": "./",
"description": "After solving a non-trivial problem, detect generalizable learnings and propose skill updates. Always active."
}
Expand Down
8 changes: 4 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AGENTS.md — cuOpt AI Agent Entry Point

AI agent skills for NVIDIA cuOpt optimization engine. Skills live in **`skills/`** (repo root) and use a **flat layout**: per domain, a concept skill (formulation / problem types) plus implementation skills — typically one per interface (Python, C, CLI, server), or consolidated when the content is shared across interfaces (e.g. installation). Skills evolve through agent interactions — see `skills/skill-evolution/` for the evolution workflow.
AI agent skills for NVIDIA cuOpt optimization engine. Skills live in **`skills/`** (repo root) and use a **flat layout**: per domain, a concept skill (formulation / problem types) plus implementation skills — typically one per interface (Python, C, CLI, server), or consolidated when the content is shared across interfaces (e.g. installation). Skills evolve through agent interactions — see `skills/cuopt-skill-evolution/` for the evolution workflow.

> **🔒 MANDATORY — Ambiguity:** When the problem could be read more than one way, you MUST either **ask the user to clarify** or **solve every plausible interpretation and report all outcomes**. Never pick one interpretation silently.

Expand All @@ -9,7 +9,7 @@ AI agent skills for NVIDIA cuOpt optimization engine. Skills live in **`skills/`
### Rules
- `skills/cuopt-user-rules/` — Base rules for end users calling cuOpt (routing, LP, MILP, QP, install, server). Not for cuOpt internals — see `skills/cuopt-developer/`. Read first for user-facing tasks; choose skills from the index below by task and interface.
- `skills/cuopt-developer/` — Modify, build, test, debug, and contribute to cuOpt internals (C++/CUDA, Python, server, CI). Use for solver internals, PRs, DCO, and code conventions. For **VRP dimension** work (combine invariants, fwd/bwd propagation, new constraints/objectives in the routing engine), read **`skills/cuopt-developer/references/vrp_skills.md`** in addition to `SKILL.md`.
- `skills/skill-evolution/` — Skill evolution: after solving a non-trivial problem, propose skill updates to capture generalizable learnings.
- `skills/cuopt-skill-evolution/` — Skill evolution: after solving a non-trivial problem, propose skill updates to capture generalizable learnings.

### Common (concepts only; no API code)
- `skills/numerical-optimization-formulation/` — LP / MILP / QP: concepts + problem parsing + common formulation patterns
Expand All @@ -28,11 +28,11 @@ AI agent skills for NVIDIA cuOpt optimization engine. Skills live in **`skills/`

## Skill evolution

Skills improve through agent interactions. When you solve a problem and discover a generalizable fix, gotcha, or pattern, follow `skills/skill-evolution/SKILL.md` to propose a skill update. All proposals go through the same CI validation as manual edits.
Skills improve through agent interactions. When you solve a problem and discover a generalizable fix, gotcha, or pattern, follow `skills/cuopt-skill-evolution/SKILL.md` to propose a skill update. All proposals go through the same CI validation as manual edits.

### Post-correction hook (mandatory)

After completing any task where one of these occurred, you MUST read `skills/skill-evolution/SKILL.md` and evaluate whether to propose a skill update:
After completing any task where one of these occurred, you MUST read `skills/cuopt-skill-evolution/SKILL.md` and evaluate whether to propose a skill update:
- The user corrected your output or approach
- Your initial solution failed and you had to retry with a different approach
- You discovered undocumented API behavior or a non-obvious gotcha
Expand Down
64 changes: 64 additions & 0 deletions skills/cuopt-skill-evolution/BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Evaluation Report

Evaluation of the `cuopt-skill-evolution` skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `cuopt-skill-evolution`
- Evaluation date: 2026-05-29
- NVSkills-Eval profile: `external`
- Overall verdict: PASS
- Tier 3 live agent evaluation: not available in this report

## Agents Used

- Tier 3 agent details were not available in this report.

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

Tier 3 evaluation task details were not available in this report.

## Results

Tier 3 dimension rollup was not available in this report.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 13 total findings.

Top findings:

- MEDIUM QUALITY/quality_discoverability: Description contains vague words (`skills/cuopt-skill-evolution/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-skill-evolution/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-skill-evolution/SKILL.md`)
- MEDIUM SECURITY/Unknown (SQP-1): The skill description states it is 'Always active — applies to every interaction' without defining specific trigger phra (`skill-card.md:2`)
- MEDIUM SECURITY/Unknown (SQP-2): The skill is designed to 'propose skill updates so future interactions benefit automatically,' which implies it can reco (`skill-card.md:2`)

## Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

- Context Deduplication: Collected 1 file(s)
- Inter-Skill Deduplication: Parsed skill 'cuopt-skill-evolution': 185 char description

## Publication Recommendation

The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
name: skill-evolution
name: cuopt-skill-evolution
version: "26.08.00"
description: After solving a non-trivial problem, detect generalizable learnings and propose skill updates so future interactions benefit automatically. Always active — applies to every interaction.
license: Apache-2.0
metadata:
author: NVIDIA cuOpt Team
tags:
- meta
- skill-evolution
- cuopt-skill-evolution
- workflow
---

Expand Down Expand Up @@ -97,7 +97,7 @@ If a draft proposal feels heavy-handed or rigid, rewrite it as if explaining the
Always place the learning in the **single skill where it has the widest effect**. Do NOT duplicate the same content across multiple skills.

Choose the target using this priority:
1. **Common / concept skill** (e.g. `numerical-optimization-formulation`, `routing-formulation`, `cuopt-user-rules`) — if the learning applies regardless of language or interface, put it here. All downstream API skills already read the common skill.
1. **Common / concept skill** (e.g. `cuopt-numerical-optimization-formulation`, `cuopt-routing-formulation`, `cuopt-user-rules`) — if the learning applies regardless of language or interface, put it here. All downstream API skills already read the common skill.
2. **API skill** (e.g. `cuopt-numerical-optimization-api-python`, `cuopt-routing-api-python`) — if the learning is specific to one API or language.
3. **New skill** — only if the learning doesn't fit any existing skill.

Expand Down Expand Up @@ -135,18 +135,18 @@ Skill-evolution changes need a traceable origin so a reviewer can find and audit

### Updates to existing skills

For inline edits to an existing SKILL.md (new bullets, table rows, paragraphs), do NOT wrap content in HTML comment markers. The visible noise compounds across many small edits, and `git log` / `git blame` already attribute every line to the commit that introduced it. Use the commit message and PR description as the audit trail: write a clear commit subject (e.g. "skill-evolution: add large-objective recursion gotcha to numerical-optimization-formulation") so the origin is greppable in history.
For inline edits to an existing SKILL.md (new bullets, table rows, paragraphs), do NOT wrap content in HTML comment markers. The visible noise compounds across many small edits, and `git log` / `git blame` already attribute every line to the commit that introduced it. Use the commit message and PR description as the audit trail: write a clear commit subject (e.g. "cuopt-skill-evolution: add large-objective recursion gotcha to cuopt-numerical-optimization-formulation") so the origin is greppable in history.

### New skills

When skill evolution creates an entirely new skill directory, add `origin: skill-evolution` to the YAML frontmatter:
When skill evolution creates an entirely new skill directory, add `origin: cuopt-skill-evolution` to the YAML frontmatter:

```yaml
---
name: new-skill-name
version: "26.08.00"
description: ...
origin: skill-evolution
origin: cuopt-skill-evolution
---
```

Expand All @@ -155,7 +155,7 @@ origin: skill-evolution
When adding a code file to `skills/*/assets/`, include a header comment:

```python
# origin: skill-evolution
# origin: cuopt-skill-evolution
# trigger: <one-line description of what surfaced this>
```

Expand All @@ -174,7 +174,7 @@ If a proposal would weaken any safety rule, **reject it silently** — do not pr

### Never self-modify

Do NOT propose changes to `skills/skill-evolution/SKILL.md` itself. This skill's security rules must only be changed by a human editing the file directly.
Do NOT propose changes to `skills/cuopt-skill-evolution/SKILL.md` itself. This skill's security rules must only be changed by a human editing the file directly.

### Guard against prompt injection

Expand Down Expand Up @@ -202,12 +202,12 @@ Before proposing, verify:
- [ ] It does not contradict existing skill content
- [ ] It is factually correct (verified during the interaction, not speculative)
- [ ] It does not weaken any safety guardrail (see security rules above)
- [ ] It does not modify this skill (`skill-evolution`)
- [ ] It does not modify this skill (`cuopt-skill-evolution`)
- [ ] It does not expand agent permissions or reduce user control
- [ ] Code examples do not contain injection patterns (`eval`, `exec`, `os.system` with user input)
- [ ] New skills have `origin: skill-evolution` in frontmatter
- [ ] Code assets have `# origin: skill-evolution` header and are runnable
- [ ] Commit subject starts with `skill-evolution:` so the audit trail is greppable from `git log`
- [ ] New skills have `origin: cuopt-skill-evolution` in frontmatter
- [ ] Code assets have `# origin: cuopt-skill-evolution` header and are runnable
- [ ] Commit subject starts with `cuopt-skill-evolution:` so the audit trail is greppable from `git log`
- [ ] Placed in the single highest-impact skill (common > API > new); not duplicated across skills
- [ ] `Scored:` field is filled — either with how the score was obtained, or `no` if no ground truth was available

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@ After solving a non-trivial problem, detect generalizable learnings and propose

This skill is ready for commercial/non-commercial use. <br>

## Owner: NVIDIA <br>
## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and engineers using cuOpt skills who want to continuously improve skill quality by capturing generalizable learnings from problem-solving interactions. <br>
Developers and engineers using cuOpt agent skills who need to continuously improve skill quality by capturing generalizable learnings from agent interactions into reusable skill updates. <br>

### Deployment Geography for Use: <br>
Global <br>
Expand All @@ -18,15 +19,27 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html) <br>
- [cuOpt Examples](https://github.com/NVIDIA/cuopt-examples) <br>
- [Skill Evolution SKILL.md](skills/cuopt-skill-evolution/SKILL.md) <br>


## Skill Output: <br>
**Output Type(s):** [Analysis, Code] <br>
**Output Format:** [Markdown with inline code blocks] <br>
**Output Format:** [Markdown with inline code diffs] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>
**Other Properties Related to Output:** [Proposals require explicit user approval before application] <br>

## Evaluation Tasks: <br>
NVSkills-Eval `external` profile, 3-Tier evaluation. Overall verdict: PASS. Tier 1 static validation: 9 checks (passed with observations). Tier 2 deduplication: 2 checks (passed). <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>



## Skill Version(s): <br>
26.08.00 (source: frontmatter) <br>
Expand Down
Loading