skills: prefix non-product-scoped skills with cuopt- - #1319
Closed
ramakrishnap-nv wants to merge 6 commits into
Closed
skills: prefix non-product-scoped skills with cuopt-#1319ramakrishnap-nv wants to merge 6 commits into
ramakrishnap-nv wants to merge 6 commits into
Conversation
The NVIDIA Agent Skills Publishing Onboarding Guide (Skill Naming & Taxonomy section) requires product-scoped skills to include the product in the skill name — to avoid collisions across products and give users context when browsing or installing from the public catalog. Three skills in this repo lacked the cuopt- prefix: skill-evolution → cuopt-skill-evolution numerical-optimization-formulation → cuopt-numerical-optimization-formulation routing-formulation → cuopt-routing-formulation PR 1310 is the catalog-onboarding prep PR, so the cuOpt skills are not yet "published" to github.com/NVIDIA/skills — the grandfather clause that lets pre-existing published skills keep their names does not yet apply. Renaming now avoids a forced rename after first publication. Changes: - git mv the 3 skill directories (carries SKILL.md, evals/, skill-card.md, skill.oms.sig with them — NVCARPS will re-sign on the next cycle). - Update SKILL.md frontmatter `name:` to match the new directory name (spec requires dir == frontmatter name). - Update evals/evals.json `expected_skill` values. - Update cross-references: AGENTS.md skill index (4 mentions), .claude-plugin/marketplace.json (3 entries), .agents/rules/skill-evolution.mdc renamed to cuopt-skill-evolution.mdc with content updated, skills/cuopt-user-rules/SKILL.md post-correction hook pointer. - Update self-references inside cuopt-skill-evolution/SKILL.md (provenance tagging examples: `origin: cuopt-skill-evolution`, commit-subject prefix, security rules). Verified: all 12 skills now satisfy dir == frontmatter name, all are kebab-case ≤64 chars, and a negative-lookbehind grep for any bare old-name reference (not preceded by `cuopt-`) returns empty across the entire repo. Non-goals: cuopt-developer is left as-is despite being contributor-facing (page 5 guidance suggests moving it to .agents/contributor-skills/ but that requires a components.d/<slug>.yml change in the catalog repo and is deferred to a follow-up). Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
|
Actionable comments posted: 0 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughRenames skills to a cuopt-prefixed namespace, updates the plugin marketplace and an agent rule to reference the new names, adjusts documentation and SKILL frontmatter, and adds/updates cuopt-specific evals and provenance/user-rule instructions. ChangescuOpt Skill Namespace Consolidation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
ramakrishnap-nv
added
non-breaking
Collaborator
Author
|
/nvskills-ci |
…ser-rules
The renames brought these two meta/behavioral skills into the eval gate
for the first time. Both regressed:
cuopt-skill-evolution: claude-code -0.02 overall, composite FAIL
cuopt-user-rules: claude-code -0.09 overall (codex carried PASS)
Root cause: same baseline-saturation pattern we hit earlier on
cuopt-server-common. The old questions tested behaviors the base model
already does (asking clarifying questions, recognizing a user
correction), so the LLM judge penalized the with-skill run for being
*more* cautious than baseline.
Rewrite each eval to test content that only the skill documents:
- cuopt-skill-evolution: the exact five-field proposal block
(Target / Trigger / Scored / Removal / Diff) in that order. Base
model invents generic field names; the skill prescribes these five
exactly.
- cuopt-user-rules: the three-element Result Summary format — solver
status drawn from {Optimal, FeasibleFound, SUCCESS}, objective value
rendered in bold or a code block (not inline prose), and a brief
description of what the objective represents. Base model writes
prose summaries that won't hit the specific format.
The eval questions are phrased naturally (no skill name mentioned in
the prompt, ground_truth, or expected_behavior) so the with-skill run
gets no free hint via the question text — it has to recover the answer
shape from the skill content. `expected_skill` still tags which skill
the framework grades.
Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
Collaborator
Author
|
/nvskills-ci |
|
Actionable comments posted: 0 |
The previous evals for cuopt-numerical-optimization-formulation and
cuopt-routing-formulation were baseline-saturated — the four-label
parsing framework and TSP/VRP/PDP identification are both textbook OR
knowledge that any capable LLM can answer cold. Codex hit ~0.97 on
both without the skill loaded, leaving no headroom and producing the
two HIGH regressions that blocked the previous CI cycle.
New evals lean on rules that ONLY each skill teaches:
- numopt: tests the committed-phrasing rule ("operates"/"employs" →
parameter, not facility/hiring decision), the MANDATORY ambiguity
rule (must surface "plans to produce A, B, C" as ambiguous instead
of silently picking), and the cost+revenue → maximize-profit rule
(baseline default is minimize-cost).
- routing: tests the required-data-categories workflow on a severely
underspecified prompt. Baseline LLMs reliably default to VRP and
start coding; the skill forces an explicit ask across all five
routing-specific data categories before formulating.
Question/ground_truth/expected_behavior text does not mention skill
names — keeps the with-skill run from getting a free routing hint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/nvskills-ci |
|
Actionable comments posted: 0 |
The first rewrite (rt-form-eval-001-underspecified-routing-data) was too strict — it demanded the agent ask about all 5 data categories before doing anything, which the LLM judge graded as "not addressing the task" and produced a FAIL verdict with claude-code overall lift -0.08 in local validation. Replace with rt-form-eval-001-pdp-with-missing-data: the prompt now gives enough specifics (8 trucks, 50 jobs, ~20 pickup-deliver pairs) that the skill can demonstrate value via PDP identification + naming specific missing data, rather than a generic ask-everything reply. The expected_behavior list is collapsed to a single focused item to reduce LLM-judge variance against saturated claude-code. Local nv-base validate result on PR 1319 branch: Verdict: NEUTRAL claude-code lift: +0.00 codex lift: +0.06 (previous CI run had codex at -0.06 — the rewrite reverses the regression that triggered one of the three HIGH gate findings) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/nvskills-ci |
|
Actionable comments posted: 0 |
…le behavior CI run on f0fd64e produced 1 HIGH finding — codex regressed -0.12 on numopt-form-eval-001 because the 4-item expected_behavior demanded an ambiguity-surfacing behavior that the LLM judge consistently fails to detect even when the skill is loaded. Local run on the same eval text hit +0.22 on codex; same eval, different judge draws → variance. Trim to a single focused behavior — the maximize-profit-not-minimize-cost rule, which is the most uniquely cuOpt-skill-driven differentiator (baseline LLMs reliably default to minimize-cost when seeing 'determine the plan + costs'). Mirrors the pattern that worked for cuopt-routing-formulation (commit f0fd64e). ground_truth still describes the full skill expectation (committed phrasing, ambiguity awareness, no code) so Accuracy / Goal Accuracy have rich context to grade against — only the strict Behavior Check list is collapsed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/nvskills-ci |
|
Actionable comments posted: 0 |
Collaborator
Author
|
/nvskills-ci |
The skill was thin (~40 lines, mostly TSP/VRP/PDP textbook content) and
had no cuOpt-specific content distinguishing it from general OR
knowledge. Empirically:
- claude-code baseline scored 1.00 every CI cycle — no headroom for
the skill to demonstrate lift.
- codex was variance-prone, contributing recurring HIGH findings to
gate_severity.py with no offsetting signal (overall lift hovering
around 0 with ±0.10 swings).
Routing-specific cuOpt conventions (depot at index 0, order-index
pickup-delivery pair encoding, capacity-array shape, cost matrix dtype)
live in cuopt-routing-api-python where they belong with the API. The
standalone formulation skill duplicated nothing useful.
Removed:
- skills/cuopt-routing-formulation/ (SKILL.md, evals, skill-card,
skill.oms.sig)
- Entry in .claude-plugin/marketplace.json
- Bullet in AGENTS.md "Common (concepts only; no API code)" section
- Reference in cuopt-skill-evolution/SKILL.md placement-rule example
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/nvskills-ci |
ramakrishnap-nv
deleted the
skills/rename-non-product-scoped-to-cuopt-prefix
branch
July 9, 2026 18:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Renames three skills to satisfy the Skill Naming & Taxonomy rule in the NVIDIA Agent Skills Publishing Onboarding Guide (product-scoped skills must include the product in the skill name to avoid collisions across products in the public catalog).