Add 7 noisy skills with iterated evals to nvidia/skills repo - #5058
Closed
Phlip79 wants to merge 4 commits into
Closed
Add 7 noisy skills with iterated evals to nvidia/skills repo#5058Phlip79 wants to merge 4 commits into
Phlip79 wants to merge 4 commits into
Conversation
This is the split-off half of PR #5036. It groups the seven skills whose Tier 3 lift evals are stochastically borderline so they can be iterated on without holding back the more stable five (cicd, onboard-gb200-1node-tests, run-on-slurm, testing, update-golden-values). Skills included: - build-and-dependency - bump-base-image - create-issue - linting-and-formatting - nightly-sync - respond-to-issue - split-pr Eval datasets use two-fact prompts (one skill-specific value plus a plain-text format directive) tuned to keep the LLM-as-Judge lift near zero for both claude-code and codex local validation runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original multi-bullet split-planning question regressed codex by
-0.08 on the most recent nvskills-ci run. Local iteration shows the
v3 pattern — two skill-specific facts plus a plain-text format
directive — produces consistent positive lift on both agents:
- claude-code: {0.000, +0.109} across two consecutive runs
- codex: {+0.052, +0.036}
The chosen facts are the git diff + git apply extraction one-liner
and the fork-vs-upstream push rule, both of which a baseline agent
tends to get wrong (cherry-pick instead of git apply; pushing directly
to NVIDIA/Megatron-LM).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pull the update-golden-values changes (SKILL.md license/destructive-op warning, evals.json, skill-card.md, skill.oms.sig) over from PR #5036 where the codex agent lift regressed on pipeline 53056542. This PR already groups skills with known judge variance, so it is the right home for it.
Phlip79
added a commit
that referenced
this pull request
May 29, 2026
Drop the update-golden-values skill changes (SKILL.md edits, evals.json, skill-card.md, skill.oms.sig) from this PR. The skill's codex agent lift regressed on pipeline 53056542 after passing in the prior five runs, so it is being relocated to the noisy-skills PR (#5058) to keep this PR stable.
wujingyue
approved these changes
May 29, 2026
This was referenced May 29, 2026
Five less-reliable skills moved to a separate PR so this one can land on its first nvskills-ci pass without being held back by the others. Skills remaining in #5058: - create-issue (CI 53047129 codex +0.04; local clean) - linting-and-formatting (CI 53047129 codex +0.07; local strong) - nightly-sync (CI 53047129 codex +0.07; local clean) Moved to philip/skills-most-flaky: - build-and-dependency, bump-base-image, respond-to-issue, split-pr, update-golden-values Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phlip79
added a commit
that referenced
this pull request
May 29, 2026
Drop the update-golden-values skill changes (SKILL.md edits, evals.json, skill-card.md, skill.oms.sig) from this PR. The skill's codex agent lift regressed on pipeline 53056542 after passing in the prior five runs, so it is being relocated to the noisy-skills PR (#5058) to keep this PR stable.
Member
Author
|
Closing in favor of #5066 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split off from #5036. This PR carries the seven skills whose Tier 3 lift evals are stochastically borderline so we can iterate on them without holding back the other five (cicd, onboard-gb200-1node-tests, run-on-slurm, testing, update-golden-values) which are now in #5036 unaffected.
Skills included here:
Eval design:
Two-fact prompts (one skill-specific value plus a plain-text format directive) tuned to keep the LLM-as-Judge lift near zero for both claude-code and codex on local validation. Tier 3 lift has structural per-run variance of roughly ±0.05 per agent per skill, so a clean local run is not a guarantee against an occasional CI HIGH finding — these skills will likely need re-runs and further iteration after the first nvskills-ci pass on this branch.
Test plan
/nvskills-cionce the branch is up🤖 Generated with Claude Code