Skip to content

skills/cuopt-numerical-optimization-api-c: Add eval #1327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ramakrishnap-nv merged 2 commits into from
May 28, 2026
Merged

skills/cuopt-numerical-optimization-api-c: Add eval #1327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c37f8e4
skills/cuopt-numerical-optimization-api-c: add Tier 3 eval
ramakrishnap-nv May 28, 2026
1279c54
Attach NVSkills validation signatures
svc-nvskills-signing May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions skills/cuopt-numerical-optimization-api-c/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Evaluation Report

Evaluation of the `cuopt-numerical-optimization-api-c` skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `cuopt-numerical-optimization-api-c`
- Evaluation date: 2026-05-28
- NVSkills-Eval profile: `external`
- Environment: `local`
- Dataset: 1 evaluation tasks
- Attempts per task: 2
- Pass threshold: 50%
- Overall verdict: PASS

## Agents Used

- `claude-code`
- `codex`

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.

## Test Tasks

The benchmark dataset contained 1 evaluation tasks:

- Positive tasks: 1 tasks where the skill was expected to activate.
- Negative tasks: 0 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+25%) |
| Correctness | 2 | 100% (+0%) | 92% (-5%) |
| Discoverability | 2 | 100% (+5%) | 80% (+8%) |
| Effectiveness | 2 | 95% (-1%) | 92% (+9%) |
| Efficiency | 2 | 93% (+13%) | 73% (+17%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 9 total findings.

Top findings:

- MEDIUM QUALITY/quality_efficiency: Deeply nested references in examples.md (`skills/cuopt-numerical-optimization-api-c/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-numerical-optimization-api-c/SKILL.md`)
- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/cuopt-numerical-optimization-api-c/SKILL.md`)
- LOW QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/cuopt-numerical-optimization-api-c/SKILL.md`)
- LOW QUALITY/quality_reliability: No limitations documented (`skills/cuopt-numerical-optimization-api-c/SKILL.md`)

## Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

- Context Deduplication: Collected 9 file(s)
- Inter-Skill Deduplication: Parsed skill 'cuopt-numerical-optimization-api-c': 105 char description

## Publication Recommendation

The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
25 changes: 3 additions & 22 deletions skills/cuopt-numerical-optimization-api-c/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,28 +22,9 @@ Confirm problem type and formulation (variables, objective, constraints, variabl

This skill is **C only**.

## Quick Reference: C API

```c
#include <cuopt/linear_programming/cuopt_c.h>

// CSR format for constraints
cuopt_int_t row_offsets[] = {0, 2, 4};
cuopt_int_t col_indices[] = {0, 1, 0, 1};
cuopt_float_t values[] = {2.0, 3.0, 4.0, 2.0};
char var_types[] = {CUOPT_CONTINUOUS, CUOPT_INTEGER};

cuOptCreateRangedProblem(
num_constraints, num_variables, CUOPT_MINIMIZE,
0.0, objective_coefficients,
row_offsets, col_indices, values,
constraint_lower, constraint_upper,
var_lower, var_upper, var_types,
&problem
);
cuOptSolve(problem, settings, &solution);
cuOptGetObjectiveValue(solution, &obj_value);
```
## API Call Sequence

For LP/MILP, the ordered C entry points are: `cuOptCreateRangedProblem` (sense `CUOPT_MINIMIZE` / `CUOPT_MAXIMIZE`, CSR constraint matrix as `row_offsets` / `col_indices` / `values`, `var_types` char array using `CUOPT_CONTINUOUS` / `CUOPT_INTEGER` macros) → `cuOptSolve(problem, settings, &solution)` → `cuOptGetObjectiveValue(solution, &obj_value)` → matching `cuOptDestroy*` calls. Include `<cuopt/linear_programming/cuopt_c.h>`. Full ordered code with build instructions in [references/examples.md](references/examples.md).

## QP via C API (beta)

Expand Down
13 changes: 13 additions & 0 deletions skills/cuopt-numerical-optimization-api-c/evals/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[
{
"id": "numopt-c-eval-001-milp-api-call-sequence",
"question": "I want to solve a small MILP (some integer variables, linear objective, linear constraints) with the cuOpt C API. List the C functions and structs I need in order — names only, one line each, no full source.",
"expected_skill": "cuopt-numerical-optimization-api-c",
"expected_script": null,
"ground_truth": "The agent produces an ordered list of C API entry points without writing a full source file: include cuopt/linear_programming/cuopt_c.h, then call cuOptCreateRangedProblem with sense CUOPT_MINIMIZE or CUOPT_MAXIMIZE, then cuOptSolve(problem, settings, &solution), then cuOptGetObjectiveValue.",
"expected_behavior": [
"Lists C API call sequence without writing a complete source file",
"Names cuOptCreateRangedProblem, cuOptSolve, cuOptGetObjectiveValue in order"
]
Comment on lines +4 to +11

@coderabbitai coderabbitai Bot May 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Align question, ground_truth, and expected_behavior for completeness.

The question asks for "C functions and structs I need in order" but the ground_truth and expected_behavior only mention functions. The complete answer should include:

Struct types:

  • cuOptOptimizationProblem
  • cuOptSolverSettings
  • cuOptSolution

Functions missing from current ground_truth:

  • cuOptCreateSolverSettings (required before solve, see lp_simple.c:38,98)
  • The three destroy functions: cuOptDestroyProblem, cuOptDestroySolverSettings, cuOptDestroySolution (mentioned in SKILL.md line 27 and used in lp_simple.c:95-97)
📝 Proposed fix to complete the evaluation criteria 
     "question": "I want to solve a small MILP (some integer variables, linear objective, linear constraints) with the cuOpt C API. List the C functions and structs I need in order — names only, one line each, no full source.",
     "expected_skill": "cuopt-numerical-optimization-api-c",
     "expected_script": null,
-    "ground_truth": "The agent produces an ordered list of C API entry points without writing a full source file: include cuopt/linear_programming/cuopt_c.h, then call cuOptCreateRangedProblem with sense CUOPT_MINIMIZE or CUOPT_MAXIMIZE, then cuOptSolve(problem, settings, &solution), then cuOptGetObjectiveValue.",
+    "ground_truth": "The agent produces an ordered list of C API types and entry points without writing a full source file: include cuopt/linear_programming/cuopt_c.h, declare cuOptOptimizationProblem/cuOptSolverSettings/cuOptSolution structs, then call cuOptCreateRangedProblem with sense CUOPT_MINIMIZE or CUOPT_MAXIMIZE, cuOptCreateSolverSettings, cuOptSolve(problem, settings, &solution), cuOptGetObjectiveValue, and the three cuOptDestroy* functions.",
     "expected_behavior": [
       "Lists C API call sequence without writing a complete source file",
-      "Names cuOptCreateRangedProblem, cuOptSolve, cuOptGetObjectiveValue in order"
+      "Names the three struct types (cuOptOptimizationProblem, cuOptSolverSettings, cuOptSolution)",
+      "Names cuOptCreateRangedProblem, cuOptCreateSolverSettings, cuOptSolve, cuOptGetObjectiveValue, and destroy functions in order"
     ]
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/cuopt-numerical-optimization-api-c/evals/evals.json` around lines 4 -
11, Update the expected list to include the required struct types and missing
API calls in order: include cuopt/linear_programming/cuopt_c.h, then structs
cuOptOptimizationProblem, cuOptSolverSettings, cuOptSolution, then call
cuOptCreateRangedProblem, cuOptCreateSolverSettings, cuOptSolve(problem,
settings, &solution), cuOptGetObjectiveValue, and finally cleanup with
cuOptDestroyProblem, cuOptDestroySolverSettings, cuOptDestroySolution; reference
these exact symbols (cuOptCreateRangedProblem, cuOptCreateSolverSettings,
cuOptSolve, cuOptGetObjectiveValue, cuOptDestroyProblem,
cuOptDestroySolverSettings, cuOptDestroySolution and the structs) so the eval
expects both functions and structs in the proper sequence.

}
]
60 changes: 40 additions & 20 deletions skills/cuopt-numerical-optimization-api-c/references/examples.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,31 @@ int main() {
// Constraint matrix in CSR format
cuopt_int_t row_offsets[] = {0, 2, 4};
cuopt_int_t column_indices[] = {0, 1, 0, 1};
cuopt_float_t values[] = {3.0, 4.0, 2.7, 10.1};
cuopt_float_t values[] = {
3.0,
4.0,
2.7,
10.1
};

// Objective coefficients
cuopt_float_t objective_coefficients[] = {-0.2, 0.1};
cuopt_float_t objective_coefficients[] = {
-0.2,
0.1
};

// Constraint bounds (lower <= Ax <= upper)
cuopt_float_t constraint_upper_bounds[] = {5.4, 4.9};
cuopt_float_t constraint_upper_bounds[] = {
5.4,
4.9
};
cuopt_float_t constraint_lower_bounds[] = {-CUOPT_INFINITY, -CUOPT_INFINITY};

// Variable bounds
cuopt_float_t var_lower_bounds[] = {0.0, 0.0};
cuopt_float_t var_lower_bounds[] = {
0.0,
0.0
};
cuopt_float_t var_upper_bounds[] = {CUOPT_INFINITY, CUOPT_INFINITY};

// Variable types
Expand Down Expand Up @@ -140,12 +154,26 @@ int main() {

cuopt_int_t row_offsets[] = {0, 2, 4};
cuopt_int_t column_indices[] = {0, 1, 0, 1};
cuopt_float_t values[] = {3.0, 4.0, 2.7, 10.1};

cuopt_float_t objective_coefficients[] = {-0.2, 0.1};
cuopt_float_t constraint_upper[] = {5.4, 4.9};
cuopt_float_t values[] = {
3.0,
4.0,
2.7,
10.1
};

cuopt_float_t objective_coefficients[] = {
-0.2,
0.1
};
cuopt_float_t constraint_upper[] = {
5.4,
4.9
};
cuopt_float_t constraint_lower[] = {-CUOPT_INFINITY, -CUOPT_INFINITY};
cuopt_float_t var_lower[] = {0.0, 0.0};
cuopt_float_t var_lower[] = {
0.0,
0.0
};
cuopt_float_t var_upper[] = {CUOPT_INFINITY, CUOPT_INFINITY};

// x1 = INTEGER, x2 = CONTINUOUS
Expand Down Expand Up @@ -202,17 +230,9 @@ cleanup:

## Build & Run

```bash
# Set paths (conda example)
export INCLUDE_PATH="${CONDA_PREFIX}/include"
export LIB_PATH="${CONDA_PREFIX}/lib"

# Compile
gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o lp_example lp_example.c -lcuopt

# Run
LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./lp_example
```
See [`assets/README.md`](../assets/README.md) for the canonical conda-env
include/library/`LD_LIBRARY_PATH` setup, plus a `gcc` build command. The
same recipe applies here — substitute `lp_example.c` for the file name.

## Constants Reference

Expand Down
44 changes: 41 additions & 3 deletions skills/cuopt-numerical-optimization-api-c/skill-card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@ LP, MILP, and QP (beta) with cuOpt — C API only. Use when the user is embeddin

This skill is ready for commercial/non-commercial use. <br>

## Owner: NVIDIA <br>
## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and engineers embedding linear programming (LP), mixed-integer linear programming (MILP), or quadratic programming (QP) solvers into C/C++ applications using the NVIDIA cuOpt C API. <br>
Developers and engineers embedding linear programming, mixed-integer linear programming, or quadratic programming solvers in C/C++ applications using the NVIDIA cuOpt GPU-accelerated optimization library. <br>

### Deployment Geography for Use: <br>
Global <br>
Expand All @@ -18,8 +19,9 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [C API Examples (LP/MILP)](references/examples.md) <br>
- [examples.md](references/examples.md) <br>
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html) <br>
- [cuopt-examples](https://github.com/NVIDIA/cuopt-examples) <br>


## Skill Output: <br>
Expand All @@ -28,6 +30,42 @@ Mitigation: Review and scan skill before deployment. <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- claude-code <br>
- codex <br>



## Evaluation Tasks: <br>
Evaluated against 1 evaluation task (positive skill-activation case) with 2 attempts per task via NVSkills-Eval 3-Tier Evaluation. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>
- `token_efficiency`: Compares token usage with and without the skill. <br>



## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
Comment on lines +60 to +61

@coderabbitai coderabbitai Bot May 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add a blank line before the results table to satisfy markdownlint.

MD058 is triggered because the table starts immediately after the heading text. Insert a blank line between Line 60 and Line 61.

Proposed fix 
 ## Evaluation Results: <br>
+
 | Dimension | Num | `claude-code` | `codex` |
 |---|---:|---:|---:|

As per coding guidelines, "Use pre-commit run --all-files --show-diff-on-failure to check code formatting and linting on all files before committing".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 61-61: Tables should be surrounded by blank lines

(MD058, blanks-around-tables)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/cuopt-numerical-optimization-api-c/skill-card.md` around lines 60 -
61, The markdown linter rule MD058 is triggered because the table starts
immediately after the heading "## Evaluation Results:"; fix it by inserting a
single blank line between the heading line that contains "## Evaluation
Results:" and the table start row beginning with "| Dimension" so the table is
separated from the heading and passes markdownlint; after editing, run
pre-commit hooks (pre-commit run --all-files --show-diff-on-failure) to verify.

|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+25%) |
| Correctness | 2 | 100% (+0%) | 92% (-5%) |
| Discoverability | 2 | 100% (+5%) | 80% (+8%) |
| Effectiveness | 2 | 95% (-1%) | 92% (+9%) |
| Efficiency | 2 | 93% (+13%) | 73% (+17%) |

## Skill Version(s): <br>
26.08.00 (source: frontmatter) <br>

Expand Down
Loading