Skip to content

skills/cuopt-numerical-optimization-api-python: update with benchmark details #1332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ramakrishnap-nv merged 2 commits into from
May 29, 2026
Merged

skills/cuopt-numerical-optimization-api-python: update with benchmark details #1332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
52c285c
skills/cuopt-numerical-optimization-api-python: update with benchmark…
ramakrishnap-nv May 29, 2026
739c9fd
Attach NVSkills validation signatures
svc-nvskills-signing May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions skills/cuopt-numerical-optimization-api-python/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Evaluation Report

Evaluation of the `cuopt-numerical-optimization-api-python` skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `cuopt-numerical-optimization-api-python`
- Evaluation date: 2026-05-29
- NVSkills-Eval profile: `external`
- Environment: `local`
- Dataset: 1 evaluation tasks
- Attempts per task: 2
- Pass threshold: 50%
- Overall verdict: FAIL

## Agents Used

- `claude-code`
- `codex`

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.

## Test Tasks

The benchmark dataset contained 1 evaluation tasks:

- Positive tasks: 1 tasks where the skill was expected to activate.
- Negative tasks: 0 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 100% (+0%) | 87% (+5%) |
| Discoverability | 2 | 100% (+0%) | 67% (+0%) |
| Effectiveness | 2 | 80% (-3%) | 54% (+4%) |
| Efficiency | 2 | 93% (-0%) | 56% (-4%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 18 total findings.

Top findings:

- MEDIUM PII/gps_coordinates: GPS coordinates (location information) (`SKILL.md:212`)
- MEDIUM PII/gps_coordinates: GPS coordinates (location information) (`references/qp_examples.md:162`)
- MEDIUM PII/gps_coordinates: GPS coordinates (location information) (`references/qp_examples.md:163`)
- MEDIUM PII/gps_coordinates: GPS coordinates (location information) (`references/qp_examples.md:164`)
- MEDIUM PII/phone_numbers: International phone number (`assets/mps_solver/results.md:48`)

## Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 9 total findings.

Top findings:

- HIGH DUPLICATE/duplicate: Duplicate content found across assets/lp_warmstart/README.md and assets/lp_warmstart/model.py:
"# LP PDLP Warmstart" in assets/lp_warmstart/README.md (lines 1-5)
vs "(module docstring)" in assets/lp_warmstart/model.py (lines 1-4) (`assets/lp_warmstart/README.md:1`)
- HIGH DUPLICATE/duplicate: Duplicate content found across SKILL.md and assets/mps_solver/README.md and references/qp_examples.md:
"# Solve" in SKILL.md (lines 63-67)
vs "# Configure and solve" in assets/mps_solver/README.md (lines 76-80)
vs "# Solve" in references/qp_examples.md (lines 47-51) (`SKILL.md:63`)
- HIGH DUPLICATE/duplicate: Duplicate content found across assets/milp_basic/README.md and assets/milp_basic/model.py:
"# Minimal MILP" in assets/milp_basic/README.md (lines 1-10)
vs "(module docstring)" in assets/milp_basic/model.py (lines 1-6) (`assets/milp_basic/README.md:1`)
- HIGH DUPLICATE/duplicate: Duplicate content found within SKILL.md:
"# MILP-specific settings" in SKILL.md (lines 94-100)
vs "# MILP gap tolerance (stop when within X% of optimal)" in SKILL.md (lines 216-218) (`SKILL.md:94`)
- HIGH DUPLICATE/duplicate: Duplicate content found across SKILL.md and assets/mps_solver/README.md:
"# Check status (CRITICAL: use PascalCase!)" in SKILL.md (lines 68-74)
vs "# ✅ CORRECT" in SKILL.md (lines 148-151)
vs "# Check solution" in assets/mps_solver/README.md (lines 81-85) (`SKILL.md:68`)

## Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.
1 change: 1 addition & 0 deletions skills/cuopt-numerical-optimization-api-python/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ metadata:
- python
---


@coderabbitai coderabbitai Bot May 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

❓ Verification inconclusive

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE="skills/cuopt-numerical-optimization-api-python/SKILL.md"

echo "== File exists and line 1-40 =="
ls -l "$FILE"
nl -ba "$FILE" | sed -n '1,60p'

echo "== Diff for the file (if any) =="
if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  git --no-pager diff -- "$FILE" || true
  echo
  echo "== Current HEAD commit message =="
  git --no-pager log -1 --format='%H%n%B' || true
else
  echo "Not a git repo in this sandbox."
fi

echo "== pre-commit config presence =="
ls -la .pre-commit-config.yaml pre-commit-config.yaml .pre-commit-config.yml pre-commit-config.yml 2>/dev/null || true

echo "== Search for documentation about /nvskills-ci =="
rg -n "/nvskills-ci" -S . || true

echo "== Search for guidance mentioning pre-commit run --all-files =="
rg -n "pre-commit run --all-files" -S . || true

echo "== pre-commit docs or CONTRIBUTING references =="
for f in README.md CONTRIBUTING.md .github/CONTRIBUTING.md .github/workflows/* 2>/dev/null; do
  if [ -f "$f" ]; then
    echo "---- $f ----"
    rg -n "pre-commit|nvskills|/nvskills-ci" -S "$f" || true
  fi
done

Repository: NVIDIA/cuopt

Length of output: 255

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE="skills/cuopt-numerical-optimization-api-python/SKILL.md"

echo "== File exists and show first 80 lines =="
ls -l "$FILE"
awk 'NR<=80 {printf "%6d\t%s\n", NR, $0}' "$FILE"

echo "== Check if this is a git repo (for HEAD commit message) =="
if command -v git >/dev/null 2>&1 && git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  echo "Git repo: yes"
  echo "== HEAD commit message (subject+body) =="
  git --no-pager log -1 --format='%s%n%n%b' || true
else
  echo "Git repo: no (or git not available)"
fi

echo "== pre-commit config presence =="
ls -la .pre-commit-config.yaml pre-commit-config.yaml .pre-commit-config.yml pre-commit-config.yml 2>/dev/null || true

echo "== Search for /nvskills-ci guidance =="
rg -n "/nvskills-ci" -S . || true

echo "== Search for guidance mentioning pre-commit run --all-files =="
rg -n "pre-commit run --all-files" -S . || true

echo "== Search for any CONTRIBUTING/README references =="
for f in README.md CONTRIBUTING.md .github/CONTRIBUTING.md .github/workflows/*; do
  if [ -f "$f" ]; then
    echo "---- $f ----"
    rg -n "pre-commit|nvskills|/nvskills-ci" -S "$f" || true
  fi
done

Repository: NVIDIA/cuopt

Length of output: 13305

Trigger NVSkills CI and show pre-commit evidence for this skills/ update
skills/cuopt-numerical-optimization-api-python/SKILL.md is under skills/ and requires: running pre-commit run --all-files --show-diff-on-failure (and committing any resulting fixes) and having /nvskills-ci commented on the PR so the nv-nvskill-ci[bot] signature commit is added and remains. The current commit already includes a Signed-off-by; please add/confirm the pre-commit and /nvskills-ci items in the PR.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/cuopt-numerical-optimization-api-python/SKILL.md` at line 16, This PR
modifies skills/cuopt-numerical-optimization-api-python/SKILL.md and must run
pre-commit across skills/ before merge: run `pre-commit run --all-files
--show-diff-on-failure`, apply and commit any fixes (ensuring updated files in
the same branch), then add a comment with `/nvskills-ci` on the PR so the
nv-nvskill-ci[bot] signature commit is produced and remains; confirm the final
commit(s) include the pre-commit changes and the bot signature.

# cuOpt Numerical Optimization Skill (Python)

Model and solve LP, MILP, and QP problems using NVIDIA cuOpt's GPU-accelerated solver. The Python API surface (`Problem`, `SolverSettings`, `solve`) is shared across all three problem classes — only the objective form and a few rules change.
Expand Down
54 changes: 46 additions & 8 deletions skills/cuopt-numerical-optimization-api-python/skill-card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
## Description: <br>
Solve Linear Programming (LP), Mixed-Integer Linear Programming (MILP), and Quadratic Programming (QP, beta) with the NVIDIA cuOpt Python API. <br>
Solve Linear Programming (LP), Mixed-Integer Linear Programming (MILP), and Quadratic Programming (QP, beta) problems with the NVIDIA cuOpt Python API. <br>

This skill is ready for commercial/non-commercial use. <br>

## Owner: NVIDIA <br>
## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 <br>
Apache-2.0 <br>
## Use Case: <br>
Developers and engineers who need to formulate and solve linear, mixed-integer linear, and quadratic optimization problems using NVIDIA cuOpt's GPU-accelerated Python API for applications such as scheduling, resource allocation, facility location, production planning, and portfolio optimization. <br>
Developers and engineers solving mathematical optimization problems (LP, MILP, QP) using the cuOpt Python API for scheduling, resource allocation, facility location, production planning, and portfolio optimization. <br>

### Deployment Geography for Use: <br>
Global <br>
Expand All @@ -18,17 +19,54 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [QP Python API Examples](references/qp_examples.md) <br>
- [QP Examples Reference](references/qp_examples.md) <br>
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html) <br>
- [cuOpt API Reference](https://docs.nvidia.com/cuopt/user-guide/latest/api.html) <br>
- [cuOpt Examples Repository](https://github.com/NVIDIA/cuopt-examples) <br>


## Skill Output: <br>
**Output Type(s):** [Code, Analysis] <br>
**Output Format:** [Markdown with inline Python code blocks] <br>
**Output Type(s):** [Code, API Calls] <br>
**Output Format:** [Python code with inline comments] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- claude-code <br>
- codex <br>



## Evaluation Tasks: <br>
Evaluated against 1 evaluation task with 2 attempts per task via NVSkills-Eval (external profile). Pass threshold: 50%. Overall verdict: FAIL. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>
- `token_efficiency`: Compares token usage with and without the skill. <br>



## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+0%) |
Comment on lines +62 to +64

@coderabbitai coderabbitai Bot May 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add a blank line before the evaluation table to satisfy markdownlint MD058.

The table starts immediately after the heading and triggers the reported lint warning.

Proposed fix 
 ## Evaluation Results: <br>
+
 | Dimension | Num | `claude-code` | `codex` |
 |---|---:|---:|---:|
 | Security | 2 | 100% (+0%) | 100% (+0%) |

As per coding guidelines Use pre-commit run --all-files --show-diff-on-failure to check code formatting and linting on all files before committing.

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 62-62: Tables should be surrounded by blank lines

(MD058, blanks-around-tables)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/cuopt-numerical-optimization-api-python/skill-card.md` around lines 62
- 64, Add a single blank line between the preceding heading and the evaluation
table that begins with the row starting "| Dimension | Num | `claude-code` |
`codex` |" in skill-card.md so the table is not immediately adjacent to the
heading (this satisfies markdownlint MD058); edit the content around the
heading/table and insert one empty line before that table to fix the lint error
and then run pre-commit (pre-commit run --all-files --show-diff-on-failure) to
verify formatting.

| Correctness | 2 | 100% (+0%) | 87% (+5%) |
| Discoverability | 2 | 100% (+0%) | 67% (+0%) |
| Effectiveness | 2 | 80% (-3%) | 54% (+4%) |
| Efficiency | 2 | 93% (-0%) | 56% (-4%) |

## Skill Version(s): <br>
26.08.00 (source: frontmatter) <br>

Expand Down
Loading