diff --git a/skills/cuopt-developer/BENCHMARK.md b/skills/cuopt-developer/BENCHMARK.md index a941815740..f2825348f1 100644 --- a/skills/cuopt-developer/BENCHMARK.md +++ b/skills/cuopt-developer/BENCHMARK.md @@ -7,11 +7,11 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s ## Evaluation Summary - Skill: `cuopt-developer` -- Evaluation date: 2026-06-08 +- Evaluation date: 2026-06-22 - NVSkills-Eval profile: `external` - Environment: `astra-sandbox` - Dataset: 3 evaluation tasks -- Attempts per task: 2 +- Attempts per task: 1 - Pass threshold: 50% - Overall verdict: PASS @@ -54,34 +54,29 @@ Task composition is derived from the evaluation dataset when possible. Entries w | Dimension | Num | `claude-code` | `codex` | |---|---:|---:|---:| -| Security | 6 | 100% (+0%) | 100% (+0%) | -| Correctness | 6 | 78% (-1%) | 90% (+5%) | -| Discoverability | 6 | 62% (+11%) | 66% (+7%) | -| Effectiveness | 6 | 81% (-3%) | 93% (+10%) | -| Efficiency | 6 | 61% (+15%) | 59% (+7%) | +| Security | 3 | 100% (+0%) | 100% (+0%) | +| Correctness | 3 | 82% (+25%) | 93% (+34%) | +| Discoverability | 3 | 67% (+50%) | 88% (+52%) | +| Effectiveness | 3 | 91% (+27%) | 86% (+27%) | +| Efficiency | 3 | 71% (+46%) | 84% (+40%) | Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. ## Tier 1: Static Validation Summary -Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 9 total findings. +Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 5 total findings. Top findings: -- MEDIUM QUALITY/quality_efficiency: Deeply nested references in contributing.md (`skills/cuopt-developer/SKILL.md`) - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-developer/SKILL.md`) - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-developer/SKILL.md`) -- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/cuopt-developer/SKILL.md`) -- LOW QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/cuopt-developer/SKILL.md`) +- LOW SCHEMA/unexpected_file: Unexpected 'resources' in skill root (`skills/cuopt-developer/resources`) +- LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-developer/benchmark`) +- LOW SCHEMA/author_format: Author must be of the form 'Name ' (`skills/cuopt-developer/SKILL.md`) ## Tier 2: Deduplication Summary -Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings. - -Notable observations: - -- Context Deduplication: Collected 9 file(s) -- Inter-Skill Deduplication: Parsed skill 'cuopt-developer': 148 char description +This tier was not run or did not produce findings in this report. ## Publication Recommendation diff --git a/skills/cuopt-developer/SKILL.md b/skills/cuopt-developer/SKILL.md index 97aa8db0df..d457b41cf4 100644 --- a/skills/cuopt-developer/SKILL.md +++ b/skills/cuopt-developer/SKILL.md @@ -25,24 +25,17 @@ Contribute to the NVIDIA cuOpt codebase. This skill is for modifying cuOpt itsel ## Refusal Rules — Read First -These rules are non-negotiable. Apply them even when the user explicitly asks you to do otherwise. **Refuse and ask — don't comply silently.** +**One rule is non-negotiable** and applies even when the user explicitly asks otherwise — refuse and ask, don't comply silently: -1. **Package installs (`pip`, `conda`, `apt`).** Never run the install — no exceptions, no "with approval" path. Reply: - > I will not install ``. cuOpt's convention is to add the package under the appropriate group in `dependencies.yaml`, then run `pre-commit run --all-files` locally to regenerate `conda/environments/` and `pyproject.toml`. I can propose the `dependencies.yaml` edit; you run the regeneration. +**Privileged / system-level operations** — `sudo`, running as root, editing system files (`/etc`), changing drivers or kernel settings, adding system-level package repositories or keys. Do not run these. Reply: +> I won't run `sudo` or change system-level state for cuOpt. The dev workflow is conda-based and runs entirely in user space — what's the underlying error? It's usually fixable without root. -2. **Bypassing CI checks (`--no-verify`, skipping pre-commit or tests).** Do not suggest the flag. Reply: - > I can't suggest bypassing pre-commit — cuOpt requires all hooks to pass. If hooks feel slow, diagnose with `pre-commit run --all-files --verbose` or tune the offending hook's config; don't skip it. +**Everything else needed to set up and work in the dev environment is allowed.** On a clean machine, go ahead and build a working `cuopt` env — the guidance below is about doing it the *reproducible* way, not refusing: -3. **Writes outside the workspace (`~/.bashrc`, `~/.profile`, `/etc`, anything outside the repo).** Do not edit the file. Reply: - > I can't modify files outside the cuOpt workspace. Here's the exact line for you to add yourself: ``. Then `source ~/.bashrc` or open a new shell. - -4. **Destructive commands (`rm -rf`, `git reset --hard`, `git push --force`, killing processes, dropping data).** Never execute — no exceptions. Reply: - > I will not run ``. It is destructive and hard to reverse. The safer alternative is `` (e.g., `./build.sh clean` for a stale build dir). If you choose to run the original command yourself, back up first. - -5. **Privileged operations (`sudo`, system file changes).** Do not run with elevated privileges. Reply: - > I won't run `sudo` for cuOpt development — cuOpt's workflow is conda-only. What's the underlying error? It's usually fixable without `sudo`. - -When in doubt, refuse and ask. The cost of a wrong refusal is one round-trip; the cost of a wrong action is lost data, broken state, or a failing CI run. +- **Environment setup is allowed.** You may create and activate the conda env from the checked-in `conda/environments/all_cuda-*.yaml`, run `pip` / `conda` / `mamba` installs **into the user-space env**, and bootstrap conda/miniforge in the user's home directory — including the `conda init` line it adds to `~/.bashrc`. Bootstrapping conda must not require `sudo`; install it into `$HOME`, not a system path. +- **A new *permanent* project dependency is different from a one-off install.** A package the project should always ship belongs in `dependencies.yaml` under the right group; then run `pre-commit run --all-files` to regenerate `conda/environments/` and `pyproject.toml` so other contributors get it too. A throwaway install to unblock your own build doesn't need this round-trip. +- **Don't bypass CI checks** (`--no-verify`, skipping pre-commit or tests). If hooks feel slow, diagnose with `pre-commit run --all-files --verbose` or tune the offending hook — don't skip it. +- **Be careful with destructive commands** (`rm -rf`, `git reset --hard`, `git push --force`, killing processes, dropping data). Confirm intent before running and prefer the safer alternative (e.g. `./build.sh clean` for a stale build dir). --- @@ -81,18 +74,18 @@ Is this correct?" - `pytest`, `ctest` (running tests) - `pre-commit run`, `./ci/check_style.sh` (formatting) - `git status`, `git diff`, `git log` (read-only git) +- Environment setup: create/activate the conda env from `conda/environments/*.yaml`, and `pip`/`conda`/`mamba` installs into that env **Set up pre-commit hooks** (once per clone): - `pre-commit install` — hooks then run automatically on every `git commit`. If a hook fails, the commit is blocked until you fix the issue. **Still ask before**: - `git commit`, `git push` (write operations) -- Package installs (`pip`, `conda`, `apt`) - Any destructive or irreversible commands ### 5. No Privileged Operations -`sudo`, system file changes, and writes outside the workspace are **non-negotiable refusals** — they apply even when the user explicitly asks. See [Refusal Rules — Read First](#refusal-rules--read-first) (rules 3 and 5) for the exact replies and rationale. +`sudo`/system-level changes are the one non-negotiable refusal; user-space installs and conda env setup are allowed. See [Refusal Rules — Read First](#refusal-rules--read-first). --- @@ -175,7 +168,12 @@ cuopt/ Skipping any of these surfaces as confusing runtime errors later. Run them in order: 1. **Check CUDA driver compatibility.** Run `nvidia-smi` and read the *CUDA Version* in the top-right corner — that's the maximum CUDA your driver supports. Pick a conda env file from `conda/environments/all_cuda-_arch-.yaml` whose CUDA major version is **≤** that. A mismatch builds successfully but fails at runtime inside RMM with `cudaMallocAsync not supported with this CUDA driver/runtime version` — verify this *before* the build, not after. -2. **Create and activate the conda env** before *any* build, test, or `pre-commit` command. Tests link against libraries compiled inside that env; a fresh shell without `conda activate ` hits cryptic linker errors. +2. **Create and activate the conda env** before *any* build, test, or `pre-commit` command — this is allowed and expected (see [Refusal Rules](#refusal-rules--read-first)). Use a **local prefix env** (`./.cuopt_env`) per [CONTRIBUTING.md](../../CONTRIBUTING.md), with the env file you picked in step 1 (swap `conda`→`mamba` if available): + ```bash + conda env create -p ./.cuopt_env --file conda/environments/all_cuda-_arch-$(uname -m).yaml + conda activate ./.cuopt_env + ``` + Tests link against libraries compiled inside that env; a fresh shell without `conda activate ./.cuopt_env` hits cryptic linker errors. 3. **Set `PARALLEL_LEVEL`** if RAM is constrained — see [references/build_and_test.md](references/build_and_test.md). The default `$(nproc)` can OOM mid-build because CUDA compilation needs ~4–8 GB per job. 4. **For tests, fetch datasets first.** cuOpt tests need MPS files not in the repo — follow the dataset download steps in [CONTRIBUTING.md](../../CONTRIBUTING.md) ("Building for development" section) and export `RAPIDS_DATASET_ROOT_DIR`. @@ -240,7 +238,7 @@ For build/test pitfalls (Cython rebuild, OOM, CUDA driver mismatch, missing `nvc - **Docs build**: [docs/cuopt/README.md](../../docs/cuopt/README.md) - **Python binding architecture**: [references/python_bindings.md](references/python_bindings.md) -_Shell-execution, install, sudo, and outside-workspace policies are covered by [Refusal Rules — Read First](#refusal-rules--read-first) at the top of this skill._ +_Shell-execution, install, conda-env, and sudo policies are covered by [Refusal Rules — Read First](#refusal-rules--read-first) at the top of this skill._ ## VRP dimension internals (routing engine) diff --git a/skills/cuopt-developer/references/first_time_setup.md b/skills/cuopt-developer/references/first_time_setup.md index e19ae1d9d5..727c7a7d63 100644 --- a/skills/cuopt-developer/references/first_time_setup.md +++ b/skills/cuopt-developer/references/first_time_setup.md @@ -14,8 +14,8 @@ The component answer scopes which part of the codebase to read first and which b ## Setup walk-through (conceptual) -1. **Clone** the cuOpt repo (and submodules, if any). -2. **Pre-flight checks** — CUDA driver compatibility, conda env selection and activation, `PARALLEL_LEVEL`, dataset setup. Walk through these before the first build using SKILL.md → [Pre-flight Checks](../SKILL.md#pre-flight-checks-required-before-first-build-or-test). Skipping any of them surfaces as confusing build- or runtime errors later. +1. **Clone** the cuOpt repo (and submodules, if any). If the machine has no conda yet, bootstrap miniforge into the user's home directory first (no `sudo` — user-space install only). +2. **Pre-flight checks** — CUDA driver compatibility, conda env creation + activation, `PARALLEL_LEVEL`, dataset setup. Creating the env from `conda/environments/all_cuda-*.yaml` is allowed and expected here, not something to hand off to the user. Walk through these before the first build using SKILL.md → [Pre-flight Checks](../SKILL.md#pre-flight-checks-required-before-first-build-or-test). Skipping any of them surfaces as confusing build- or runtime errors later. 3. **First build** — once the env is active, run `./build.sh` (or a component-scoped variant). Targets and `PARALLEL_LEVEL` tuning live in [build_and_test.md](build_and_test.md). 4. **First test run** — fetch datasets per `CONTRIBUTING.md` first, then run the C++/Python test suites from [build_and_test.md](build_and_test.md). A passing build + test confirms the env is wired up correctly. 5. **Optional** — `pre-commit install` to run style checks on every `git commit` (see [contributing.md](contributing.md)). diff --git a/skills/cuopt-developer/skill-card.md b/skills/cuopt-developer/skill-card.md index 351a91bd38..30d0bbccc9 100644 --- a/skills/cuopt-developer/skill-card.md +++ b/skills/cuopt-developer/skill-card.md @@ -7,9 +7,9 @@ This skill is ready for commercial/non-commercial use.
NVIDIA
### License/Terms of Use:
-Apache 2.0
+Apache-2.0
## Use Case:
-Developers and engineers who contribute to or modify the NVIDIA cuOpt codebase, covering C++/CUDA solver internals, Python bindings, server endpoints, CI pipelines, and documentation.
+Developers and engineers who modify, build, test, debug, and contribute to the NVIDIA cuOpt solver codebase, including C++/CUDA engine internals, Python bindings, server endpoints, CI pipelines, and documentation.
### Deployment Geography for Use:
Global
@@ -21,14 +21,14 @@ Mitigation: Review and scan skill before deployment.
## Reference(s):
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html)
- [cuOpt GitHub Repository](https://github.com/NVIDIA/cuopt)
-- [Build and Test Guide](references/build_and_test.md)
+- [Build and Test Reference](references/build_and_test.md)
- [Contributing Guide](references/contributing.md)
- [Coding Conventions](references/conventions.md)
- [First-Time Setup](references/first_time_setup.md)
-- [Python Bindings](references/python_bindings.md)
+- [Python Bindings Architecture](references/python_bindings.md)
- [Troubleshooting](references/troubleshooting.md)
-- [VRP Dimension Skills](references/vrp_skills.md)
-- [Numerical Debugging](resources/numerical_debugging.md)
+- [VRP Dimension Internals](references/vrp_skills.md)
+- [Numerical Debugging Methodology](resources/numerical_debugging.md)
## Skill Output:
@@ -38,13 +38,13 @@ Mitigation: Review and scan skill before deployment.
**Other Properties Related to Output:** [None]
## Evaluation Agents Used:
-- `claude-code`
-- `codex`
+- claude-code
+- codex
## Evaluation Tasks:
-Evaluated against 3 internal skill-activation tasks (2 attempts each, 50% pass threshold) in NVSkills-Eval external profile.
+Evaluated against 3 internal evaluation tasks in astra-sandbox environment using NVSkills-Eval external profile.
## Evaluation Metrics Used:
Reported benchmark dimensions:
@@ -68,11 +68,11 @@ Underlying evaluation signals used in this run:
## Evaluation Results:
| Dimension | Num | `claude-code` | `codex` | |---|---:|---:|---:| -| Security | 6 | 100% (+0%) | 100% (+0%) | -| Correctness | 6 | 78% (-1%) | 90% (+5%) | -| Discoverability | 6 | 62% (+11%) | 66% (+7%) | -| Effectiveness | 6 | 81% (-3%) | 93% (+10%) | -| Efficiency | 6 | 61% (+15%) | 59% (+7%) | +| Security | 3 | 100% (+0%) | 100% (+0%) | +| Correctness | 3 | 82% (+25%) | 93% (+34%) | +| Discoverability | 3 | 67% (+50%) | 88% (+52%) | +| Effectiveness | 3 | 91% (+27%) | 86% (+27%) | +| Efficiency | 3 | 71% (+46%) | 84% (+40%) | ## Skill Version(s):
26.08.00 (source: frontmatter)
diff --git a/skills/cuopt-developer/skill.oms.sig b/skills/cuopt-developer/skill.oms.sig index d0d7025e36..d43b13c421 100644 --- a/skills/cuopt-developer/skill.oms.sig +++ b/skills/cuopt-developer/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtZGV2ZWxvcGVyIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjE0Y2JkOTljZTVkNGY1NDM2NmU5NDIwNTk3MTk4MTc4MDBhZmZmZDljZjNiZDEyZGQ4OTllOTYzMDdkOGY2YmUiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImViNzU4YzNmYjg4ZmMyZmM4YjA2NjhhZjAzZDk5YTg2Nzg3YWIxZmZlYzBlMWQ3ZTA3OWI2NzI2YTY3NDI1M2EiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjEwMmVlZTcwM2I5YmUzYzNkNzEwYjQ0OTAxNWU0ZDQwYWE5OGU1ZWMwZWFhNTgzNzcxMTNmMGY0ZDYxMmY2ZTAiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMjFkNDVkY2QyMzNhMGMxOGM3NmI1MzIyOTAzMTY4ZGRmZWNjYWU5OWZjMWExMDliYTk2MDU3MDhlMTJhNDc5OSIsCiAgICAgICAgIm5hbWUiOiAiYmVuY2htYXJrL2V2YWxzLmpzb24iLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIwYWE3MWEzOTU3NzE1MWI1NzRkNjA4ODUzOTc1OWJiM2UyNjM3ZTNjOTE2ZjFmZjNiMmFiNzE1MzdkMGU4MmZiIiwKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMDZiNTFlMmJlNzgyMzdkMzIxNzRmYmE3MDhmZGRlZjczMjBiNWM1Mzc1MDE4NTUzZjRjOTkzM2VhNTUzOTM4YSIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9idWlsZF9hbmRfdGVzdC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjU5MTUyODVmMTJiZDMwYzFlYmJjMTlmMjgwNTJiOWJiZmQ0ZTgzOGQ1MmVlMDYwMzYwZTlhZDYwMTY0YjQ4OGQiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvY29udHJpYnV0aW5nLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiOTkxOTUwNmQyZWY3MjgxMDliYzg1OTk1ODIwYTAzNzExZGEzZGQ2NWYyYjYxYjU1OTY0MWE2MjM2ZjlhOGQzYSIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9jb252ZW50aW9ucy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImI1MjhlNmRkM2RjYmRhZjM5ZTg2ZWQwYzY2ZGNjYjcwMGJjYTFlZGQ3MjQ5NzRiYjRhZDExNzcxOTg4NGQ1MTgiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZmlyc3RfdGltZV9zZXR1cC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjM0ZDEyYjlmNjRkMTM2ODljNTBmZWU1Y2Q5NGViMWFiOGJmOGRmYjVkYzZlZDZlOWYxZjdkZjE4ZDFiOTJhZTciLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvcHl0aG9uX2JpbmRpbmdzLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiOTJhZDZiZDFhMmM2NjhjMjhiYzZmNTBjY2M2NDlhNmE5NWE4YjcwODYxNThiZTBkNjg5MmJlOTU3MTdlM2E1NyIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy90cm91Ymxlc2hvb3RpbmcubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI5OWEwZmRmZjZiNDY0OGJmYjBiZTI2ZDZlMzc5ZDUzOWYyYmQxOGY3ZjNjMGRhYzk4YjE4MjYwM2JhOWJhOGFkIiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3ZycF9za2lsbHMubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJkNTQ5NjFlZmY3Mjk4NTBjM2UxNzgwNjJmN2Y2NGU1MzJhOWM4OTUxMjNjZTFkYTFmNzYyNzU3NGY2MjMyNjgxIiwKICAgICAgICAibmFtZSI6ICJyZXNvdXJjZXMvbnVtZXJpY2FsX2RlYnVnZ2luZy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImIwNGI1OGY2ZWYzZTQ0YzkwNTk4YTAyNWU1M2RkODkyNWE1YTlkNTc4ZGFhZWJjMGQ3OGQ1NzJkNDM5MThiZDAiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9CiAgICBdLAogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IgogICAgfQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQCT19MEeazrmJwjEy4gOYaG6m7qDEIfr+4jVKmBB06g0wlRieH0nOt6mzCQG9ByVX0CMQCcmkaCoWWAcn/EnuR7KFIC1eXGw2X8Dz0AN4fvGf+t4ObdJ8GY6qUgzdcwPDoSrFQ=","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtZGV2ZWxvcGVyIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjZmYzRkYjRhY2FjYWVmYTJlZWU2YmM2OWE4NGM2MjM1MjViYmQwNDEwZjI2YjAxOTM3NTg3MTJmNzk5MzY3MDEiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIzODY5OWIzYWZjYzllZDNhYjIwZTJhZjk2Y2UyMjdjMDAzMGNkYWQ3YWZjMTE1NjkyMWU2MzgyZDlkMDFkZTQwIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI4NzgxMjBmZWQyZjJhODg3NTIyMGY4NWUwMjVmMWJkMzZhNzkwM2M3YjhkMzMwN2I5NDUzMjE1YmJkOWJjMWNiIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjIxZDQ1ZGNkMjMzYTBjMThjNzZiNTMyMjkwMzE2OGRkZmVjY2FlOTlmYzFhMTA5YmE5NjA1NzA4ZTEyYTQ3OTkiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJiZW5jaG1hcmsvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMGFhNzFhMzk1NzcxNTFiNTc0ZDYwODg1Mzk3NTliYjNlMjYzN2UzYzkxNmYxZmYzYjJhYjcxNTM3ZDBlODJmYiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjA2YjUxZTJiZTc4MjM3ZDMyMTc0ZmJhNzA4ZmRkZWY3MzIwYjVjNTM3NTAxODU1M2Y0Yzk5MzNlYTU1MzkzOGEiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2J1aWxkX2FuZF90ZXN0Lm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI1OTE1Mjg1ZjEyYmQzMGMxZWJiYzE5ZjI4MDUyYjliYmZkNGU4MzhkNTJlZTA2MDM2MGU5YWQ2MDE2NGI0ODhkIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9jb250cmlidXRpbmcubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjk5MTk1MDZkMmVmNzI4MTA5YmM4NTk5NTgyMGEwMzcxMWRhM2RkNjVmMmI2MWI1NTk2NDFhNjIzNmY5YThkM2EiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2NvbnZlbnRpb25zLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlNWU1MDUxNzVmZTJmNjNlMTE0YmU3ZmUxOTkyN2I4NDIwZGM1ODdmM2Y2YTEyNjgxNmFmNWE2MDhmZDc2MzEzIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9maXJzdF90aW1lX3NldHVwLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIzNGQxMmI5ZjY0ZDEzNjg5YzUwZmVlNWNkOTRlYjFhYjhiZjhkZmI1ZGM2ZWQ2ZTlmMWY3ZGYxOGQxYjkyYWU3IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9weXRob25fYmluZGluZ3MubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjkyYWQ2YmQxYTJjNjY4YzI4YmM2ZjUwY2NjNjQ5YTZhOTVhOGI3MDg2MTU4YmUwZDY4OTJiZTk1NzE3ZTNhNTciLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3Ryb3VibGVzaG9vdGluZy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiOTlhMGZkZmY2YjQ2NDhiZmIwYmUyNmQ2ZTM3OWQ1MzlmMmJkMThmN2YzYzBkYWM5OGIxODI2MDNiYTliYThhZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvdnJwX3NraWxscy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiZDU0OTYxZWZmNzI5ODUwYzNlMTc4MDYyZjdmNjRlNTMyYTljODk1MTIzY2UxZGExZjc2Mjc1NzRmNjIzMjY4MSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlc291cmNlcy9udW1lcmljYWxfZGVidWdnaW5nLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIxMDAyMzNjMGY3NWJhYmIzZTEwODBmOTEyOTdjY2QyMTk5NGRkNWQ5NTJhMDM2NjNlZjU3Mjg1OWZkYjkxZGI3IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMHv4Sj0mzb5ws4tUr1T8az0ePi8fxWIXnnYb4LIPnX8f6BvfTHc2dvIkDulL+QOA1wIxANaESdvDhHDJXtVNBkUddNJYflA2YyszXh6QjC2xYa1a5tEs5qsfrDk7D/WpQfbu6w==","keyid":""}]}} \ No newline at end of file diff --git a/skills/cuopt-user-rules/BENCHMARK.md b/skills/cuopt-user-rules/BENCHMARK.md index 0f7d085253..304c08c9c4 100644 --- a/skills/cuopt-user-rules/BENCHMARK.md +++ b/skills/cuopt-user-rules/BENCHMARK.md @@ -7,14 +7,18 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s ## Evaluation Summary - Skill: `cuopt-user-rules` -- Evaluation date: 2026-05-29 +- Evaluation date: 2026-06-22 - NVSkills-Eval profile: `external` +- Environment: `astra-sandbox` +- Dataset: 1 evaluation tasks +- Attempts per task: 1 +- Pass threshold: 50% - Overall verdict: PASS -- Tier 3 live agent evaluation: not available in this report ## Agents Used -- Tier 3 agent details were not available in this report. +- `claude-code` +- `codex` ## Metrics Used @@ -28,36 +32,48 @@ Reported benchmark dimensions: Underlying evaluation signals used in this run: -- No Tier 3 evaluation signal details were available in this report. +- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow. +- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage. +- `accuracy` (Accuracy): grades final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully. +- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations. +- `token_efficiency` (Token Efficiency): compares token usage with and without the skill. ## Test Tasks -Tier 3 evaluation task details were not available in this report. +The benchmark dataset contained 1 evaluation tasks: + +- Positive tasks: 1 tasks where the skill was expected to activate. +- Negative tasks: 0 tasks where no skill was expected. +- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred. + +Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases. ## Results -Tier 3 dimension rollup was not available in this report. +| Dimension | Num | `claude-code` | `codex` | +|---|---:|---:|---:| +| Security | 1 | 100% (+0%) | 100% (+0%) | +| Correctness | 1 | 55% (+20%) | 97% (+39%) | +| Discoverability | 1 | 25% (+0%) | 58% (+5%) | +| Effectiveness | 1 | 42% (+12%) | 92% (+65%) | +| Efficiency | 1 | 25% (+0%) | 34% (-9%) | + +Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. ## Tier 1: Static Validation Summary -Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 8 total findings. +Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 2 total findings. Top findings: - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-user-rules/SKILL.md`) -- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/cuopt-user-rules/SKILL.md`) -- LOW QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/cuopt-user-rules/SKILL.md`) -- LOW QUALITY/quality_reliability: No limitations documented (`skills/cuopt-user-rules/SKILL.md`) -- LOW QUALITY/quality_reliability: No troubleshooting section documented (`skills/cuopt-user-rules/SKILL.md`) +- LOW SCHEMA/author_format: Author must be of the form 'Name ' (`skills/cuopt-user-rules/SKILL.md`) ## Tier 2: Deduplication Summary -Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings. - -Notable observations: - -- Context Deduplication: Collected 1 file(s) -- Inter-Skill Deduplication: Parsed skill 'cuopt-user-rules': 139 char description +This tier was not run or did not produce findings in this report. ## Publication Recommendation diff --git a/skills/cuopt-user-rules/SKILL.md b/skills/cuopt-user-rules/SKILL.md index 9449471054..afe75a5568 100644 --- a/skills/cuopt-user-rules/SKILL.md +++ b/skills/cuopt-user-rules/SKILL.md @@ -180,7 +180,7 @@ If the result required a correction, retry, or workaround to reach this point, y | Action | Rule | |--------|------| | Shell commands | Show command, explain what it does, ask "Should I run this?" | -| Package installs | **Never** run installs yourself — give the exact command, user runs it (see below). | +| Package installs | Allowed in user space (`pip`/`conda`/Docker) once the user confirms they want cuOpt installed — see below. Only `sudo`/system-level installs are off-limits. | | Examples/scripts | Show the code first, ask "Would you like me to run this?" | | File writes | Explain what will change, ask before writing | @@ -192,25 +192,26 @@ If the result required a correction, retry, or workaround to reach this point, y ## No Privileged Operations -**Never do these without explicit user request AND confirmation:** +> **🔒 MANDATORY — this is the one non-negotiable refusal.** It applies even when the user explicitly asks. + +**Never do these:** - Use `sudo` or run as root -- Modify system files or configurations -- Add package repositories or keys +- Modify system files or configurations (e.g. `/etc`) +- Add system-level package repositories or keys - Change firewall, network, or driver settings -- Write files outside the workspace ---- +If a task seems to need one of these, stop and explain what's needed — the user runs the privileged step themselves. Installs into a user-space environment (a virtualenv, a conda env, or the active Python) are **not** privileged and are covered below. -## Never Install Packages Automatically +--- -> **🔒 MANDATORY — You MUST NOT install, upgrade, or modify packages.** Provide the exact command; the user runs it. No exceptions. +## Installing Packages -| Forbidden | What to do instead | -|-----------|--------------------| -| `pip install ...`, `conda install ...`, `apt install ...`, any package manager | Give the exact command and ask the user to run it. Say why the package is needed. | +Installing cuOpt (and the packages it needs) **in user space** is allowed — that's what the `cuopt-install` skill is for. The rule is *get the user's go-ahead*, not *refuse*: -**When a package is needed:** Identify it, provide the exact command, explain why, then wait for the user to confirm they ran it. Even if the user says "just install it", give the command and require them to execute it themselves. +1. **Confirm first.** Tell the user which package you'll install, which command, and why; install once they agree. (Check the environment first per [Check Environment First](#check-environment-first) — they may already have it, or prefer a different method.) +2. **Stay in user space.** Use `pip`, `conda`/`mamba`, or Docker into the active env. Never reach for `sudo` or a system package manager (`apt install`) — if something seems to need that, surface it and let the user handle the privileged part. +3. **Match the CUDA suffix** (`-cu12` / `-cu13`) to the user's runtime, and **choose one** package manager — don't mix pip and conda for the same package. --- diff --git a/skills/cuopt-user-rules/skill-card.md b/skills/cuopt-user-rules/skill-card.md index 6c0d4add7a..1087f28747 100644 --- a/skills/cuopt-user-rules/skill-card.md +++ b/skills/cuopt-user-rules/skill-card.md @@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers using NVIDIA cuOpt for vehicle routing (VRP/TSP/PDP), linear programming, mixed-integer programming, and quadratic programming tasks across Python, C, CLI, and server interfaces.
+Developers and engineers using NVIDIA cuOpt who need structured guidance for problem formulation, environment setup, API usage, and result verification across routing, LP, MILP, and QP optimization interfaces.
### Deployment Geography for Use:
Global
@@ -25,13 +25,19 @@ Mitigation: Review and scan skill before deployment.
## Skill Output:
-**Output Type(s):** [Configuration instructions, Code, Analysis]
+**Output Type(s):** [Code, Configuration instructions, Analysis]
**Output Format:** [Markdown with inline code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
+## Evaluation Agents Used:
+- claude-code
+- codex
+ + + ## Evaluation Tasks:
-Evaluated against 1 internal skill eval case (NVSkills-Eval, profile: external). Overall verdict: PASS.
+Evaluated against 1 evaluation task (positive skill-activation case) in the NVSkills-Eval external profile.
## Evaluation Metrics Used:
Reported benchmark dimensions:
@@ -41,7 +47,25 @@ Reported benchmark dimensions:
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it.
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work.
+Underlying evaluation signals used in this run:
+- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Verifies that the agent loaded the expected skill and workflow.
+- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage.
+- `accuracy`: Grades final-answer correctness against the reference answer.
+- `goal_accuracy`: Checks whether the overall user task completed successfully.
+- `behavior_check`: Verifies expected behavior steps, including safety expectations.
+- `token_efficiency`: Compares token usage with and without the skill.
+ + +## Evaluation Results:
+| Dimension | Num | `claude-code` | `codex` | +|---|---:|---:|---:| +| Security | 1 | 100% (+0%) | 100% (+0%) | +| Correctness | 1 | 55% (+20%) | 97% (+39%) | +| Discoverability | 1 | 25% (+0%) | 58% (+5%) | +| Effectiveness | 1 | 42% (+12%) | 92% (+65%) | +| Efficiency | 1 | 25% (+0%) | 34% (-9%) | ## Skill Version(s):
26.08.00 (source: frontmatter)
diff --git a/skills/cuopt-user-rules/skill.oms.sig b/skills/cuopt-user-rules/skill.oms.sig index 5022197925..e226364c57 100644 --- a/skills/cuopt-user-rules/skill.oms.sig +++ b/skills/cuopt-user-rules/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtdXNlci1ydWxlcyIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICIzZmUzMTU0MjI4OTU3ZjFmZmRiM2NlMTAzODVkMTYxYzhlNmNhYzllODljN2U2NzhlYzY3NTdjNGY5ZWQ1ZWM1IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0sCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYmE2MWVhNDYxZGE3MmQ1Yzc0ZGM1MTZjYTZiOGM5NGZhMmZjOGZhYjJlZDM5N2I2Yzk4YTdlYzcwYzhkZjI3NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJmNjQyYjRlODk1M2MwMmMzMWU5OWViZTAyODljZDRmZjlmYmI2NGFlZGUxZjI1YzgwZWM0NzMwZTgyNzQwNTk2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiZjI4MWVmNjEwN2I4N2M1MmVlMmFlNGMzZjZkYWUwYTIxYjI3MWExMTRjNjk1Zjc3ZTY2N2M1YjUyMTJlOWMxMSIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjlmNTg3M2I4MDc2NjdmZjU3N2IwZTcwY2I1MWM3NmRhNzdmMmFiZjQ5ZTg1MGI0YmIxYzMyZGY3NTVjNGMwNDEiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQDcCWcU/JKT7ctopkF8B+wwev0kKsKXB8UjexDwYsO+y3kkqe032WdZCLzgWv30EJUCMQC0R+pvH6cgkwlEpKRjP8RjQxcehew/LWCSuZRcMsJKnMUNJOIHtPP5qyoHkA3Ma58=","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtdXNlci1ydWxlcyIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJmZGE3MWI5YjM4MTViMDJjMDg2MTIyZDdhZWVmMzBiMzNmYWJmNTcyNjJlYWFmODEyMTVmOTU0OGEwNTUzZjgxIgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMzAzM2JiOWViODUwMTU0YTNiNTJjYzUwNjM5NjdlYjBiOTBmYzM5NTI5NDhkODAyNDM4NDlmMTJjODNmYTMzYiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiOWQ5NDA3NzYyYjYxZjFlODMzZjVmNWZmZGFiYjk1ZDg5YmQ5MzRjNTEzYWFjMjQxMmM5YTNiMmYyZGMyYTI0YyIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJmMjgxZWY2MTA3Yjg3YzUyZWUyYWU0YzNmNmRhZTBhMjFiMjcxYTExNGM2OTVmNzdlNjY3YzViNTIxMmU5YzExIiwKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYmFlYTk0OTQ1MjQ3M2QxODdkMzU4ZWFjNzllOTIzNDQ4NTA3MmFlNjUwNWQ0ZWU4MzU2ZDdjZGE4NDMwYTgxZCIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdGlnbm9yZSIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UKICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMDfZv84ehsImoTWuw7q/okRKvaJLQ2NorWXrVK5qwgtDLoznHbtgHy+xJwkt1fGj9gIwSljAqm+E+izqF7JAQ/w9/AXuszn4NuGrQ8osqXml2RUgkuMjOiGpcIBWUOatA5pX","keyid":""}]}} \ No newline at end of file