Add NeMo Evaluator skills (3 skills) to catalog - #1
Merged
Conversation
Added NVIDIA-NeMo/Evaluator with 3 skills: launching evaluations, accessing MLflow results, and NEL assistant. Catalog now covers 27 skills across 5 NVIDIA product repos. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
mosheabr
force-pushed
the
add-nemo-evaluator
branch
from
March 31, 2026 16:55
558b945 to
c1d79c1
Compare
This was referenced Apr 7, 2026
3 tasks
mosheabr
added a commit
that referenced
this pull request
May 28, 2026
PR #109 flattened components.d/deepstream.yml to put DeepStream skills at top-level: skills/deepstream-dev/ and skills/deepstream-import-vision-model/. The next sync (PR #110) correctly mirrored content into those flat locations. However, the previous nested skills/deepstream/ directory (with the same two skills under a 'deepstream' parent dir) was left behind by the sync — anti-pattern #1 in the catalog-pr-reviewer skill: the sync workflow doesn't auto-clean directories that are no longer referenced by any components.d/<slug>.yml entry. Removing it here to eliminate the duplicate and complete the flat-layout cutover. Authoritative content now lives at: - skills/deepstream-dev/ - skills/deepstream-import-vision-model/ Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
mosheabr
added a commit
that referenced
this pull request
May 29, 2026
Atomic cutover for cuOpt to flat layout. Two changes: 1. components.d/cuopt.yml restructured to 12-entry flat layout (one entry per skill, lowercase catalog_dir matching skill name). Source names were renamed by Ramakrishnan in NVIDIA/cuopt — 3 previously unprefixed skills (numerical-optimization-formulation, routing-formulation, skill-evolution) now carry the cuopt- prefix, re-signed via /nvskills-ci. 2. skills/cuopt/ orphan directory removed. After the next sync, 12 cuOpt skills land directly at skills/cuopt-<name>/ at top level. The old nested directory is no longer referenced by any components.d entry and would otherwise persist as orphan content (anti-pattern #1). Source compliance (NVIDIA/cuopt as of 2026-05-28): - 12 skill-card.md ✓ - 12 skill.oms.sig ✓ - 13 evals.json (one additional eval beyond the per-skill set) - 6 BENCHMARK.md (6/12 currently have Tier 3 eval; rest land on next per-skill /nvskills-ci runs) Catalog path changes externally — anyone with bookmarks pointing at skills/cuopt/<name>/ will need to repoint at skills/cuopt-<name>/. SPDX header added per convention adopted 2026-05-28. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
3 tasks
mosheabr
added a commit
that referenced
this pull request
May 29, 2026
PR #151 migrated NeMo MBridge to flat layout: the new components.d/nemo-mbridge.yml registers 20 skills at flat top-level paths (skills/nemo-mbridge-*/) and the old components.d/megatron-bridge.yml sweep entry was deleted. The previous catalog content under skills/Megatron-Bridge/ (a single nested skill mbridge-recipe-recommender from the legacy layout) was left behind by the sync — anti-pattern #1: sync writes content for referenced paths but does not garbage-collect directories that are no longer referenced by any components.d entry. Removing it here completes the migration. Authoritative content for NeMo MBridge now lives at skills/nemo-mbridge-*/ once the next sync runs, with no duplicate or stale path. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
mosheabr
added a commit
that referenced
this pull request
Jun 4, 2026
#2 (MED) — gh issue create can skip tracker on bad assignee: The "Track dropped skills" step has continue-on-error: true, so a bad assignee wouldn't kill the whole sync — but it would skip tracker creation entirely, defeating the feature. Both branches (edit existing / create new) now follow the same tolerant pattern: create-without-assignees first, then loop --add-assignee in a trim + per-handle warning. A typo in CATALOG_TRACKER_ASSIGNEES no longer blocks the tracker from existing. #3 (LOW) — Hardcoded personal fallback: Reworked per Codex's catch that GitHub issue assignees are users, not teams — so the earlier "use a team handle" comment was wrong. If CATALOG_TRACKER_ASSIGNEES is unset we now skip assignment entirely and emit a ::warning title=Drift notifications inactive::. No personal fallback; a deactivated handle would silently mute notifications. The warning surfaces in the workflow log every run until the secret is set. #4 (LOW / clarification) — Deleted-sig case: Added an inline comment in the drift loop noting that an rsync that deletes skill.oms.sig alongside content edits registers the deletion in `git diff` (sig_changed matches → drift NOT flagged here), and the now-sigless skill falls through to the compliance step under "missing artifacts: skill.oms.sig". Defense in depth. #1 (HIGH per Sayali) — README version inconsistency on revert: NOT addressed in this commit — verified against .github/scripts/regenerate-readme.sh on main: the README table is now "Product | Description | Skills" (PR #215, commit 1104de0 on 2026-06-01 dropped the Source + Version columns). VERSIONS_FILE is declared but never consumed, and /tmp/sync-versions.txt is vestigial. So the exact "README will lie" risk Sayali raised is stale. Plan to send her the context separately and open a small follow-up PR to drop the dead sync-versions write + VERSIONS_FILE declaration so this trap doesn't catch a future reader. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
mosheabr
added a commit
that referenced
this pull request
Jun 8, 2026
#2 (MED) — gh issue create can skip tracker on bad assignee: The "Track dropped skills" step has continue-on-error: true, so a bad assignee wouldn't kill the whole sync — but it would skip tracker creation entirely, defeating the feature. Both branches (edit existing / create new) now follow the same tolerant pattern: create-without-assignees first, then loop --add-assignee in a trim + per-handle warning. A typo in CATALOG_TRACKER_ASSIGNEES no longer blocks the tracker from existing. #3 (LOW) — Hardcoded personal fallback: Reworked per Codex's catch that GitHub issue assignees are users, not teams — so the earlier "use a team handle" comment was wrong. If CATALOG_TRACKER_ASSIGNEES is unset we now skip assignment entirely and emit a ::warning title=Drift notifications inactive::. No personal fallback; a deactivated handle would silently mute notifications. The warning surfaces in the workflow log every run until the secret is set. #4 (LOW / clarification) — Deleted-sig case: Added an inline comment in the drift loop noting that an rsync that deletes skill.oms.sig alongside content edits registers the deletion in `git diff` (sig_changed matches → drift NOT flagged here), and the now-sigless skill falls through to the compliance step under "missing artifacts: skill.oms.sig". Defense in depth. #1 (HIGH per Sayali) — README version inconsistency on revert: NOT addressed in this commit — verified against .github/scripts/regenerate-readme.sh on main: the README table is now "Product | Description | Skills" (PR #215, commit 1104de0 on 2026-06-01 dropped the Source + Version columns). VERSIONS_FILE is declared but never consumed, and /tmp/sync-versions.txt is vestigial. So the exact "README will lie" risk Sayali raised is stale. Plan to send her the context separately and open a small follow-up PR to drop the dead sync-versions write + VERSIONS_FILE declaration so this trap doesn't catch a future reader. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
mosheabr
added a commit
that referenced
this pull request
Jun 8, 2026
#2 (MED) — gh issue create can skip tracker on bad assignee: The "Track dropped skills" step has continue-on-error: true, so a bad assignee wouldn't kill the whole sync — but it would skip tracker creation entirely, defeating the feature. Both branches (edit existing / create new) now follow the same tolerant pattern: create-without-assignees first, then loop --add-assignee in a trim + per-handle warning. A typo in CATALOG_TRACKER_ASSIGNEES no longer blocks the tracker from existing. #3 (LOW) — Hardcoded personal fallback: Reworked per Codex's catch that GitHub issue assignees are users, not teams — so the earlier "use a team handle" comment was wrong. If CATALOG_TRACKER_ASSIGNEES is unset we now skip assignment entirely and emit a ::warning title=Drift notifications inactive::. No personal fallback; a deactivated handle would silently mute notifications. The warning surfaces in the workflow log every run until the secret is set. #4 (LOW / clarification) — Deleted-sig case: Added an inline comment in the drift loop noting that an rsync that deletes skill.oms.sig alongside content edits registers the deletion in `git diff` (sig_changed matches → drift NOT flagged here), and the now-sigless skill falls through to the compliance step under "missing artifacts: skill.oms.sig". Defense in depth. #1 (HIGH per Sayali) — README version inconsistency on revert: NOT addressed in this commit — verified against .github/scripts/regenerate-readme.sh on main: the README table is now "Product | Description | Skills" (PR #215, commit 1104de0 on 2026-06-01 dropped the Source + Version columns). VERSIONS_FILE is declared but never consumed, and /tmp/sync-versions.txt is vestigial. So the exact "README will lie" risk Sayali raised is stale. Plan to send her the context separately and open a small follow-up PR to drop the dead sync-versions write + VERSIONS_FILE declaration so this trap doesn't catch a future reader. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
melo-gonzo
pushed a commit
to melo-gonzo/NVIDIAskills
that referenced
this pull request
Aug 5, 2026
NVIDIA#2 (MED) — gh issue create can skip tracker on bad assignee: The "Track dropped skills" step has continue-on-error: true, so a bad assignee wouldn't kill the whole sync — but it would skip tracker creation entirely, defeating the feature. Both branches (edit existing / create new) now follow the same tolerant pattern: create-without-assignees first, then loop --add-assignee in a trim + per-handle warning. A typo in CATALOG_TRACKER_ASSIGNEES no longer blocks the tracker from existing. NVIDIA#3 (LOW) — Hardcoded personal fallback: Reworked per Codex's catch that GitHub issue assignees are users, not teams — so the earlier "use a team handle" comment was wrong. If CATALOG_TRACKER_ASSIGNEES is unset we now skip assignment entirely and emit a ::warning title=Drift notifications inactive::. No personal fallback; a deactivated handle would silently mute notifications. The warning surfaces in the workflow log every run until the secret is set. NVIDIA#4 (LOW / clarification) — Deleted-sig case: Added an inline comment in the drift loop noting that an rsync that deletes skill.oms.sig alongside content edits registers the deletion in `git diff` (sig_changed matches → drift NOT flagged here), and the now-sigless skill falls through to the compliance step under "missing artifacts: skill.oms.sig". Defense in depth. NVIDIA#1 (HIGH per Sayali) — README version inconsistency on revert: NOT addressed in this commit — verified against .github/scripts/regenerate-readme.sh on main: the README table is now "Product | Description | Skills" (PR NVIDIA#215, commit 1104de0 on 2026-06-01 dropped the Source + Version columns). VERSIONS_FILE is declared but never consumed, and /tmp/sync-versions.txt is vestigial. So the exact "README will lie" risk Sayali raised is stale. Plan to send her the context separately and open a small follow-up PR to drop the dead sync-versions write + VERSIONS_FILE declaration so this trap doesn't catch a future reader. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com> Signed-off-by: Carmelo Gonzales <43048528+melo-gonzo@users.noreply.github.com>
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code