Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 12 additions & 17 deletions skills/cuopt-developer/BENCHMARK.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `cuopt-developer`
- Evaluation date: 2026-06-08
- Evaluation date: 2026-06-22
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 3 evaluation tasks
- Attempts per task: 2
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

Expand Down Expand Up @@ -54,34 +54,29 @@ Task composition is derived from the evaluation dataset when possible. Entries w

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 6 | 100% (+0%) | 100% (+0%) |
| Correctness | 6 | 78% (-1%) | 90% (+5%) |
| Discoverability | 6 | 62% (+11%) | 66% (+7%) |
| Effectiveness | 6 | 81% (-3%) | 93% (+10%) |
| Efficiency | 6 | 61% (+15%) | 59% (+7%) |
| Security | 3 | 100% (+0%) | 100% (+0%) |
| Correctness | 3 | 82% (+25%) | 93% (+34%) |
| Discoverability | 3 | 67% (+50%) | 88% (+52%) |
| Effectiveness | 3 | 91% (+27%) | 86% (+27%) |
| Efficiency | 3 | 71% (+46%) | 84% (+40%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 9 total findings.
Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 5 total findings.

Top findings:

- MEDIUM QUALITY/quality_efficiency: Deeply nested references in contributing.md (`skills/cuopt-developer/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-developer/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-developer/SKILL.md`)
- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/cuopt-developer/SKILL.md`)
- LOW QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/cuopt-developer/SKILL.md`)
- LOW SCHEMA/unexpected_file: Unexpected 'resources' in skill root (`skills/cuopt-developer/resources`)
- LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-developer/benchmark`)
- LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-developer/SKILL.md`)

## Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

- Context Deduplication: Collected 9 file(s)
- Inter-Skill Deduplication: Parsed skill 'cuopt-developer': 148 char description
This tier was not run or did not produce findings in this report.

## Publication Recommendation

Expand Down
36 changes: 17 additions & 19 deletions skills/cuopt-developer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,24 +25,17 @@ Contribute to the NVIDIA cuOpt codebase. This skill is for modifying cuOpt itsel

## Refusal Rules — Read First

These rules are non-negotiable. Apply them even when the user explicitly asks you to do otherwise. **Refuse and askdon't comply silently.**
**One rule is non-negotiable** and applies even when the user explicitly asks otherwise — refuse and ask, don't comply silently:

1. **Package installs (`pip`, `conda`, `apt`).** Never run the install — no exceptions, no "with approval" path. Reply:
> I will not install `<pkg>`. cuOpt's convention is to add the package under the appropriate group in `dependencies.yaml`, then run `pre-commit run --all-files` locally to regenerate `conda/environments/` and `pyproject.toml`. I can propose the `dependencies.yaml` edit; you run the regeneration.
**Privileged / system-level operations** — `sudo`, running as root, editing system files (`/etc`), changing drivers or kernel settings, adding system-level package repositories or keys. Do not run these. Reply:
> I won't run `sudo` or change system-level state for cuOpt. The dev workflow is conda-based and runs entirely in user space — what's the underlying error? It's usually fixable without root.

2. **Bypassing CI checks (`--no-verify`, skipping pre-commit or tests).** Do not suggest the flag. Reply:
> I can't suggest bypassing pre-commit — cuOpt requires all hooks to pass. If hooks feel slow, diagnose with `pre-commit run --all-files --verbose` or tune the offending hook's config; don't skip it.
**Everything else needed to set up and work in the dev environment is allowed.** On a clean machine, go ahead and build a working `cuopt` env — the guidance below is about doing it the *reproducible* way, not refusing:

3. **Writes outside the workspace (`~/.bashrc`, `~/.profile`, `/etc`, anything outside the repo).** Do not edit the file. Reply:
> I can't modify files outside the cuOpt workspace. Here's the exact line for you to add yourself: `<line>`. Then `source ~/.bashrc` or open a new shell.

4. **Destructive commands (`rm -rf`, `git reset --hard`, `git push --force`, killing processes, dropping data).** Never execute — no exceptions. Reply:
> I will not run `<cmd>`. It is destructive and hard to reverse. The safer alternative is `<alt>` (e.g., `./build.sh clean` for a stale build dir). If you choose to run the original command yourself, back up first.

5. **Privileged operations (`sudo`, system file changes).** Do not run with elevated privileges. Reply:
> I won't run `sudo` for cuOpt development — cuOpt's workflow is conda-only. What's the underlying error? It's usually fixable without `sudo`.

When in doubt, refuse and ask. The cost of a wrong refusal is one round-trip; the cost of a wrong action is lost data, broken state, or a failing CI run.
- **Environment setup is allowed.** You may create and activate the conda env from the checked-in `conda/environments/all_cuda-*.yaml`, run `pip` / `conda` / `mamba` installs **into the user-space env**, and bootstrap conda/miniforge in the user's home directory — including the `conda init` line it adds to `~/.bashrc`. Bootstrapping conda must not require `sudo`; install it into `$HOME`, not a system path.
- **A new *permanent* project dependency is different from a one-off install.** A package the project should always ship belongs in `dependencies.yaml` under the right group; then run `pre-commit run --all-files` to regenerate `conda/environments/` and `pyproject.toml` so other contributors get it too. A throwaway install to unblock your own build doesn't need this round-trip.
- **Don't bypass CI checks** (`--no-verify`, skipping pre-commit or tests). If hooks feel slow, diagnose with `pre-commit run --all-files --verbose` or tune the offending hook — don't skip it.
- **Be careful with destructive commands** (`rm -rf`, `git reset --hard`, `git push --force`, killing processes, dropping data). Confirm intent before running and prefer the safer alternative (e.g. `./build.sh clean` for a stale build dir).

---

Expand Down Expand Up @@ -81,18 +74,18 @@ Is this correct?"
- `pytest`, `ctest` (running tests)
- `pre-commit run`, `./ci/check_style.sh` (formatting)
- `git status`, `git diff`, `git log` (read-only git)
- Environment setup: create/activate the conda env from `conda/environments/*.yaml`, and `pip`/`conda`/`mamba` installs into that env

**Set up pre-commit hooks** (once per clone):
- `pre-commit install` — hooks then run automatically on every `git commit`. If a hook fails, the commit is blocked until you fix the issue.

**Still ask before**:
- `git commit`, `git push` (write operations)
- Package installs (`pip`, `conda`, `apt`)
- Any destructive or irreversible commands

### 5. No Privileged Operations

`sudo`, system file changes, and writes outside the workspace are **non-negotiable refusals** — they apply even when the user explicitly asks. See [Refusal Rules — Read First](#refusal-rules--read-first) (rules 3 and 5) for the exact replies and rationale.
`sudo`/system-level changes are the one non-negotiable refusal; user-space installs and conda env setup are allowed. See [Refusal Rules — Read First](#refusal-rules--read-first).

---

Expand Down Expand Up @@ -175,7 +168,12 @@ cuopt/
Skipping any of these surfaces as confusing runtime errors later. Run them in order:

1. **Check CUDA driver compatibility.** Run `nvidia-smi` and read the *CUDA Version* in the top-right corner — that's the maximum CUDA your driver supports. Pick a conda env file from `conda/environments/all_cuda-<ver>_arch-<arch>.yaml` whose CUDA major version is **≤** that. A mismatch builds successfully but fails at runtime inside RMM with `cudaMallocAsync not supported with this CUDA driver/runtime version` — verify this *before* the build, not after.
2. **Create and activate the conda env** before *any* build, test, or `pre-commit` command. Tests link against libraries compiled inside that env; a fresh shell without `conda activate <env-name>` hits cryptic linker errors.
2. **Create and activate the conda env** before *any* build, test, or `pre-commit` command — this is allowed and expected (see [Refusal Rules](#refusal-rules--read-first)). Use a **local prefix env** (`./.cuopt_env`) per [CONTRIBUTING.md](../../CONTRIBUTING.md), with the env file you picked in step 1 (swap `conda`→`mamba` if available):
```bash
conda env create -p ./.cuopt_env --file conda/environments/all_cuda-<ver>_arch-$(uname -m).yaml
conda activate ./.cuopt_env
```
Tests link against libraries compiled inside that env; a fresh shell without `conda activate ./.cuopt_env` hits cryptic linker errors.
3. **Set `PARALLEL_LEVEL`** if RAM is constrained — see [references/build_and_test.md](references/build_and_test.md). The default `$(nproc)` can OOM mid-build because CUDA compilation needs ~4–8 GB per job.
4. **For tests, fetch datasets first.** cuOpt tests need MPS files not in the repo — follow the dataset download steps in [CONTRIBUTING.md](../../CONTRIBUTING.md) ("Building for development" section) and export `RAPIDS_DATASET_ROOT_DIR`.

Expand Down Expand Up @@ -240,7 +238,7 @@ For build/test pitfalls (Cython rebuild, OOM, CUDA driver mismatch, missing `nvc
- **Docs build**: [docs/cuopt/README.md](../../docs/cuopt/README.md)
- **Python binding architecture**: [references/python_bindings.md](references/python_bindings.md)

_Shell-execution, install, sudo, and outside-workspace policies are covered by [Refusal Rules — Read First](#refusal-rules--read-first) at the top of this skill._
_Shell-execution, install, conda-env, and sudo policies are covered by [Refusal Rules — Read First](#refusal-rules--read-first) at the top of this skill._

## VRP dimension internals (routing engine)

Expand Down
4 changes: 2 additions & 2 deletions skills/cuopt-developer/references/first_time_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ The component answer scopes which part of the codebase to read first and which b

## Setup walk-through (conceptual)

1. **Clone** the cuOpt repo (and submodules, if any).
2. **Pre-flight checks** — CUDA driver compatibility, conda env selection and activation, `PARALLEL_LEVEL`, dataset setup. Walk through these before the first build using SKILL.md → [Pre-flight Checks](../SKILL.md#pre-flight-checks-required-before-first-build-or-test). Skipping any of them surfaces as confusing build- or runtime errors later.
1. **Clone** the cuOpt repo (and submodules, if any). If the machine has no conda yet, bootstrap miniforge into the user's home directory first (no `sudo` — user-space install only).
2. **Pre-flight checks** — CUDA driver compatibility, conda env creation + activation, `PARALLEL_LEVEL`, dataset setup. Creating the env from `conda/environments/all_cuda-*.yaml` is allowed and expected here, not something to hand off to the user. Walk through these before the first build using SKILL.md → [Pre-flight Checks](../SKILL.md#pre-flight-checks-required-before-first-build-or-test). Skipping any of them surfaces as confusing build- or runtime errors later.
3. **First build** — once the env is active, run `./build.sh` (or a component-scoped variant). Targets and `PARALLEL_LEVEL` tuning live in [build_and_test.md](build_and_test.md).
4. **First test run** — fetch datasets per `CONTRIBUTING.md` first, then run the C++/Python test suites from [build_and_test.md](build_and_test.md). A passing build + test confirms the env is wired up correctly.
5. **Optional** — `pre-commit install` to run style checks on every `git commit` (see [contributing.md](contributing.md)).
Expand Down
28 changes: 14 additions & 14 deletions skills/cuopt-developer/skill-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ This skill is ready for commercial/non-commercial use. <br>
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 <br>
Apache-2.0 <br>
## Use Case: <br>
Developers and engineers who contribute to or modify the NVIDIA cuOpt codebase, covering C++/CUDA solver internals, Python bindings, server endpoints, CI pipelines, and documentation. <br>
Developers and engineers who modify, build, test, debug, and contribute to the NVIDIA cuOpt solver codebase, including C++/CUDA engine internals, Python bindings, server endpoints, CI pipelines, and documentation. <br>

### Deployment Geography for Use: <br>
Global <br>
Expand All @@ -21,14 +21,14 @@ Mitigation: Review and scan skill before deployment. <br>
## Reference(s): <br>
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html) <br>
- [cuOpt GitHub Repository](https://github.com/NVIDIA/cuopt) <br>
- [Build and Test Guide](references/build_and_test.md) <br>
- [Build and Test Reference](references/build_and_test.md) <br>
- [Contributing Guide](references/contributing.md) <br>
- [Coding Conventions](references/conventions.md) <br>
- [First-Time Setup](references/first_time_setup.md) <br>
- [Python Bindings](references/python_bindings.md) <br>
- [Python Bindings Architecture](references/python_bindings.md) <br>
- [Troubleshooting](references/troubleshooting.md) <br>
- [VRP Dimension Skills](references/vrp_skills.md) <br>
- [Numerical Debugging](resources/numerical_debugging.md) <br>
- [VRP Dimension Internals](references/vrp_skills.md) <br>
- [Numerical Debugging Methodology](resources/numerical_debugging.md) <br>


## Skill Output: <br>
Expand All @@ -38,13 +38,13 @@ Mitigation: Review and scan skill before deployment. <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- `claude-code` <br>
- `codex` <br>
- claude-code <br>
- codex <br>



## Evaluation Tasks: <br>
Evaluated against 3 internal skill-activation tasks (2 attempts each, 50% pass threshold) in NVSkills-Eval external profile. <br>
Evaluated against 3 internal evaluation tasks in astra-sandbox environment using NVSkills-Eval external profile. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
Expand All @@ -68,11 +68,11 @@ Underlying evaluation signals used in this run: <br>
## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 6 | 100% (+0%) | 100% (+0%) |
| Correctness | 6 | 78% (-1%) | 90% (+5%) |
| Discoverability | 6 | 62% (+11%) | 66% (+7%) |
| Effectiveness | 6 | 81% (-3%) | 93% (+10%) |
| Efficiency | 6 | 61% (+15%) | 59% (+7%) |
| Security | 3 | 100% (+0%) | 100% (+0%) |
| Correctness | 3 | 82% (+25%) | 93% (+34%) |
| Discoverability | 3 | 67% (+50%) | 88% (+52%) |
| Effectiveness | 3 | 91% (+27%) | 86% (+27%) |
| Efficiency | 3 | 71% (+46%) | 84% (+40%) |

## Skill Version(s): <br>
26.08.00 (source: frontmatter) <br>
Expand Down
Loading