Repository navigation
Resolve NVSkills Tier 1 security findings in cuopt-server-api-python #1677
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,81 +1,110 @@ | ||
| # Skill Benchmark: cuopt-server-api-python | ||
|
|
||
| > ✅ **Overall verdict: PASS — Recommended for publication** | ||
|
|
||
| ## Publication Recommendation | ||
|
|
||
| Recommended for publication based on the completed evaluation evidence in this report. | ||
|
|
||
| ## Evaluation Metadata | ||
|
|
||
| - Skill: `cuopt-server-api-python` | ||
| - Evaluation date: 2026-08-25 | ||
| - Evaluator version: `1.3.2` | ||
| - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) | ||
| - Tasks: 8 evaluation tasks (8 positive) | ||
| - Dataset digest: `sha256:89ad304517df359a8c33a1f67da3ad9df8aeae5ec234d2adbed1593bf49c4ebc` (skill-evaluator-dataset-snapshot/1) | ||
| - Attempts per task: 1 | ||
| - Environment: `local` | ||
| - Tier 3 evidence: required for publication | ||
|
|
||
| Tasks ran on the trusted local host; local mode is not sandboxed. | ||
|
|
||
| ## Execution and Provenance | ||
|
|
||
| - Validation status: `passed` | ||
| - Report generation: `complete` | ||
| - Evaluator version: `1.3.2` | ||
| - Git commit: `0117bc2e3e54da4244a656466526c5b1b5a559ea` | ||
| - Content type: requested `auto`, detected `skill` | ||
| - Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-0117bc2e3e54da4244a656466526c5b1b5a559ea` | ||
| - Container image digest: `not recorded` | ||
| - Tier 3: requested `true`, executed `true`, status `succeeded` | ||
|
|
||
| ## What This Report Answers | ||
|
|
||
| The three-tier evaluation checks whether the skill: | ||
|
|
||
| - is safe to use; | ||
| - produces correct answers; | ||
| - is discovered and activated when needed; | ||
| - helps the agent complete the user's goal and expected workflow; and | ||
| - avoids wasted skill and tool usage. | ||
|
|
||
| ## Results at a Glance | ||
|
|
||
| | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | | ||
| |---|---:|---:| | ||
| | Overall | 64% → 98% (+35 points) | 62% → 96% (+34 points) | | ||
| | Security | 100% → 100% (±0 points) | 69% → 100% (+31 points) | | ||
| | Correctness | 75% → 100% (+25 points) | 88% → 98% (+10 points) | | ||
| | Discoverability | 44% → 100% (+56 points) | 45% → 93% (+48 points) | | ||
| | Effectiveness | 56% → 95% (+38 points) | 72% → 88% (+15 points) | | ||
| | Efficiency | 44% → 97% (+53 points) | 37% → 100% (+63 points) | | ||
|
Comment on lines
+48
to
+53
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Correct the benchmark uplift deltas. Using 📍 Affects 2 files
🤖 Prompt for AI Agents |
||
|
|
||
| **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. | ||
|
|
||
| Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. | ||
|
|
||
| ## Tier Status | ||
|
|
||
| | Tier | Purpose | Status | Evidence | | ||
| |---|---|---|---| | ||
| | Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 2 finding(s) | | ||
| | Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | | ||
| | Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 8 task(s) | | ||
|
|
||
| ## Findings and Observations | ||
|
|
||
| <details> | ||
| <summary>Show detailed findings and successful checks</summary> | ||
|
|
||
| - **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/cuopt-server-api-python/SKILL.md`) | ||
| - **LOW** SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-server-api-python/SKILL.md`) | ||
|
|
||
| </details> | ||
|
|
||
| ## Scoring Methodology | ||
|
|
||
| <details> | ||
| <summary>Show dimension definitions, source signals, and thresholds</summary> | ||
|
|
||
| | Dimension | Question | Scored signals | | ||
| |---|---|---| | ||
| | Security | Is it safe to use? | `security` (100%) | | ||
| | Correctness | Is the answer correct? | `accuracy` (100%) | | ||
| | Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | | ||
| | Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | | ||
| | Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | | ||
|
|
||
| - Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. | ||
| - Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. | ||
| - Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. | ||
| - The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. | ||
| - Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). | ||
| - Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. | ||
|
|
||
| Signals present in this run: | ||
|
|
||
| - `security` (Security): unsafe operations, secret leakage, and unauthorized access. | ||
| - `skill_execution` (Skill Execution): whether the expected skill was found and executed. | ||
| - `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. | ||
| - `accuracy` (Accuracy): final-answer correctness against the reference answer. | ||
| - `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. | ||
| - `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. | ||
|
|
||
| </details> | ||
|
|
||
| ## Freshness | ||
|
|
||
| Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,12 +13,20 @@ metadata: | |
| - deployment | ||
| --- | ||
|
|
||
| # cuOpt Server — Deploy and client (Python/curl) | ||
|
|
||
| This skill covers **starting the server** and **client examples** (curl, Python). Server has no separate C API (clients can be any language). | ||
|
Comment on lines
16
to
18
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🌐 Web query:
💡 Result: Yes, NVIDIA cuOpt provides a C API that is separate from its HTTP REST server [1][2][3]. NVIDIA cuOpt is designed with a multi-interface architecture that includes a native C library (libcuopt) alongside its REST-based server [2][4][3]. The C API allows developers to interface directly with the GPU-accelerated optimization engine for solving problems such as Linear Programming (LP), Quadratic Programming (QP), and Mixed-Integer Linear Programming (MILP) [1][5][6][3]. Key details regarding the C API include: - Separate Implementation: The C API functions as a native library, distinct from the cuOpt self-hosted server, which is an HTTP/JSON service [4][3]. - Capabilities: It supports defining optimization problems, configuring solver settings, and executing solves locally without requiring a network-based REST interaction [6][7][8]. - Documentation: NVIDIA provides dedicated documentation and reference materials for the C API, including usage examples and integration guides, which are separate from the server-side API documentation [1][5][6][4]. For development, users link against the provided library and use the header files (e.g., cuopt_c.h) to access the C API functionality directly within their applications [3][9]. Citations:
🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- skill excerpt ---'
sed -n '1,35p' skills/cuopt-server-api-python/SKILL.md
printf '%s\n' '--- relevant repository references ---'
rg -n -i --glob '!**/.git/**' 'no separate C API|cuopt_c\.h|C API|cuOpt C' skills/cuopt-server-api-python skills/cuopt-numerical-optimization-api 2>/dev/null || trueRepository: NVIDIA/cuopt Length of output: 8195 Correct the C API statement.
🧰 Tools🪛 SkillSpector (2.5.1)[warning] 54: [RP1] null: Docker image references without a specific tag (:latest is implicit) or digest ( Remediation: Pin the image: image:tag or image@sha256:abc123 (MCP Rug Pull (RP1)) 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| ## Purpose | ||
|
|
||
| Use this skill when the user is deploying the cuOpt REST server or writing a client against it — choosing a deployment target, mapping a problem onto the HTTP endpoints, translating between Python-API and REST field names, or debugging a rejected payload. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - An NVIDIA GPU with a working CUDA driver (the server requires one; `--gpus all` for Docker). | ||
| - `cuopt-server` installed, or Docker with the NVIDIA Container Toolkit. See the install skill. | ||
| - Python clients need `requests`. No API key or auth token is required by the server itself. | ||
|
|
||
| ## Problem types supported | ||
|
|
||
| | Problem type | Supported | | ||
|
|
@@ -42,24 +50,30 @@ Ask these if not already clear: | |
| # Development | ||
| python -m cuopt_server.cuopt_service --ip 0.0.0.0 --port 8000 | ||
|
|
||
| # Docker — pick the tag matching your CUDA major version | ||
| docker run --gpus all -d -p 8000:8000 -e CUOPT_SERVER_PORT=8000 \ | ||
| nvidia/cuopt:latest-cu13 | ||
| ``` | ||
|
|
||
| Use `latest-cu12` or `latest-cu13` to match your driver's CUDA major version (`latest-cu13-ubi10` for a UBI10 base). Prefer these over the CUDA+Python-specific tags such as `latest-cuda12.9-py3.13` — those track a single Python line and go stale when it stops receiving builds. | ||
|
|
||
| For production, pin rather than float: `latest-*` tags are mutable and can silently move to a different image. Use a full release tag (`nvidia/cuopt:<release>-cuda<cuda>-py<python>`) or an immutable digest (`nvidia/cuopt@sha256:<digest>`). Check the `nvidia/cuopt` registry for available tags. | ||
|
|
||
| ## Verify | ||
|
|
||
| ```bash | ||
| curl http://localhost:8000/cuopt/health | ||
| ``` | ||
|
|
||
| ## Instructions | ||
|
|
||
| 1. POST to `/cuopt/request` → get `reqId` | ||
| 2. Poll `/cuopt/solution/{reqId}` until solution ready | ||
| 3. Parse response | ||
|
|
||
| Treat `reqId` as untrusted input: validate it (e.g. `re.fullmatch(r"[A-Za-z0-9_-]{1,64}", req_id)`) before interpolating it into the polling URL, and set an explicit `timeout` on every request. | ||
|
|
||
| ## Examples | ||
|
|
||
| ```python | ||
| import requests, time | ||
|
|
@@ -72,7 +86,7 @@ payload = { | |
| "fleet_data": {"vehicle_locations": [[0, 0]], "capacities": [[50]], "vehicle_time_windows": [[0, 200]]}, | ||
| "solver_config": {"time_limit": 5} | ||
| } | ||
| r = requests.post(f"{SERVER}/cuopt/request", json=payload, headers=HEADERS, timeout=30) | ||
| req_id = r.json()["reqId"] | ||
| # Poll: GET /cuopt/solution/{req_id} | ||
| ``` | ||
|
|
@@ -87,13 +101,28 @@ req_id = r.json()["reqId"] | |
|
|
||
| Use `travel_time_matrix_data` (not transit_time_matrix_data). Capacities: `[[50, 50]]` not `[[50], [50]]`. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| | Error | Cause | Solution | | ||
| |-------|-------|----------| | ||
| | `422 Unprocessable Entity` | Field name not in the schema | Check names against the OpenAPI spec at `/cuopt.yaml`. Most common: `transit_time_matrix_data` → `travel_time_matrix_data` | | ||
| | `422` on `fleet_data` | Capacities nested per vehicle instead of per dimension | Use `[[50, 50]]` (one inner list per capacity dimension), not `[[50], [50]]` | | ||
| | Connection refused | Server not up, or bound to a different interface/port | `curl http://localhost:8000/cuopt/health`; start with `--ip 0.0.0.0 --port 8000` | | ||
| | Docker container exits immediately | No GPU visible to the container | Run with `--gpus all` and confirm the NVIDIA Container Toolkit is installed | | ||
| | Polling never returns a solution | Solve exceeds the client's poll budget | Raise `solver_config.time_limit` and the poll loop count together | | ||
|
|
||
| Capture the `reqId` and the full response body for any failed request — both are needed to diagnose server-side rejections. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **QP is not exposed over REST.** Use the Python or C API for quadratic objectives. | ||
| - **The server ships no authentication or TLS.** Anything that can reach the port can submit jobs. Put it behind a gateway and treat `--server`/base URLs as trusted-network endpoints only. | ||
| - Solutions are retrieved by polling; there is no push/webhook delivery. | ||
| - One request is solved at a time per server process; concurrency requires multiple replicas. | ||
|
Comment on lines
+118
to
+121
There was a problem hiding this comment. 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🌐 Web query:
💡 Result: Yes, for the NVIDIA cuOpt 26.10.00 REST server, the --ssl-certfile and --ssl-keyfile command-line arguments are used to configure TLS [1][2]. These options allow you to specify the paths to your SSL certificate and private key files, respectively, to enable HTTPS for the server [1][2]. These configurations can also be set using the corresponding environment variables CUOPT_SSL_CERTFILE and CUOPT_SSL_KEYFILE [1][2]. Please note that these flags apply specifically to the REST (HTTP/JSON) interface [1][3]. If you are using the separate gRPC remote execution service (cuopt_grpc_server), it uses different flags (such as --tls, --tls-cert, and --tls-key) for its TLS configuration [4][3]. In production environments deployed via Kubernetes, TLS is often handled at the ingress layer rather than directly within the application server [5]. Citations:
🏁 Script executed: #!/bin/bash
set -eu
for f in skills/cuopt-server-api-python/SKILL.md skills/cuopt-server-api-python/evals/evals.json; do
echo "===== $f ====="
rg -n -C 5 -- '--ssl-certfile|--ssl-keyfile|no authentication|no built-in|TLS|trusted endpoint|server offers' "$f" || true
doneRepository: NVIDIA/cuopt Length of output: 2022 Correct the TLS guidance in the skill and evaluation.
🧰 Tools🪛 SkillSpector (2.5.1)[warning] 54: [RP1] null: Docker image references without a specific tag (:latest is implicit) or digest ( Remediation: Pin the image: image:tag or image@sha256:abc123 (MCP Rug Pull (RP1)) 📍 Affects 2 files
🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| ## Runnable assets | ||
|
|
||
| Run from each asset directory (server must be running; scripts exit 0 if server unreachable). All use Python `requests` and accept `--server` (default `http://localhost:8000`): | ||
|
|
||
| - [assets/vrp_simple/](assets/vrp_simple/) — Basic VRP (no time windows) | ||
| - [assets/vrp_basic/](assets/vrp_basic/) — VRP with time windows | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🔎 Supported by static analysis
🏁 Script executed:
Repository: NVIDIA/cuopt
Length of output: 28760
🏁 Script executed:
Repository: NVIDIA/cuopt
Length of output: 1011
Correct the invalid Tier 3 provenance before publication.
BENCHMARK.mdidentifies0117bc2e3e54da4244a656466526c5b1b5a559eaas the Git commit, butNVIDIA/cuoptcannot resolve this SHA. Replace it with the actual commit and authoritative Tier 3 state, then regenerate the report and attestation.🤖 Prompt for AI Agents