Skip to content

fix(skills): clear error message for stale skills.sh index entries - #3261

Closed
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/skills-sh-stale-index-error-3259
Closed

Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/skills-sh-stale-index-error-3259

Conversation

@Mibayy

@Mibayy Mibayy commented Mar 26, 2026

Copy link
Copy Markdown

Closes #3259

Problem

The skills.sh index contains entries (vercel-react-best-practices, react-email, react-pdf, etc.) that appear in search results with high install counts but whose GitHub files no longer exist. When a user tries to install one, they get:

Error: Could not fetch 'skills-sh/vercel-labs/agent-skills/vercel-react-best-practices' from any source.

This looks like a user error (typo, wrong identifier) but is actually an index synchronisation issue on skills.sh's side — the skill was removed or renamed by its author.

Root cause

_resolve_source_meta_and_bundle() already distinguishes the two cases:

  • meta != None, bundle == None → index hit, GitHub 404 (stale entry)
  • meta == None, bundle == None → unknown identifier

But do_install used the same generic error message for both.

Changes

hermes_cli/skills_hub.py

do_install — split the error branch:

  • Stale entry (meta found, no bundle): named error explaining the index entry is stale, that the skill may have been renamed or removed, and that this is not a user error.
  • Unknown identifier (no meta, no bundle): original generic "Could not fetch" message preserved.

do_search — add a caveat footnote when results include skills.sh entries, explaining that the index may contain entries whose GitHub files no longer exist, and what to expect if install fails.

Before / After

Before:

Fetching: skills-sh/vercel-labs/agent-skills/vercel-react-best-practices
Error: Could not fetch 'skills-sh/vercel-labs/agent-skills/vercel-react-best-practices' from any source.

After:

Fetching: skills-sh/vercel-labs/agent-skills/vercel-react-best-practices
Error: 'skills-sh/vercel-labs/agent-skills/vercel-react-best-practices' appears in the skills-sh
index but the skill files no longer exist in the underlying repository (GitHub returned 404).
This is a stale index entry. The skill may have been renamed or removed by its author.

And in search results containing skills.sh entries:

Note: 3 result(s) come from the skills.sh index, which may contain entries whose GitHub files
have since been removed or renamed. If installation fails with a 'stale index entry' error,
the skill no longer exists at its listed path.

Tests

4 new unit tests:

  • test_do_install_stale_index_shows_helpful_message — meta found, bundle None → stale message
  • test_do_install_unknown_identifier_shows_generic_message — both None → generic message, no stale
  • test_do_search_shows_skills_sh_caveat — results with skills-sh source → caveat shown
  • test_do_search_no_caveat_for_official_only — official-only results → no caveat

Closes NousResearch#3259

When a skill appears in the skills.sh search index but its GitHub files
no longer exist (renamed, deleted, or moved by the author), the user
previously received only a generic 'Could not fetch' error with no
explanation.

Changes:
- do_install: when _resolve_source_meta_and_bundle returns metadata
  (index hit) but no bundle (GitHub 404), show a 'stale index entry'
  message explaining the file no longer exists at its listed path and
  may have been renamed or removed. The generic 'Could not fetch' message
  is preserved for unknown identifiers where even the index has no record.
- do_search: when results include entries from the skills.sh source,
  append a dim Note explaining that the skills.sh index may contain
  entries whose GitHub files have since been removed, so a 'stale index
  entry' error during install means the skill is gone -- not a user error.
  Official-only results are not annotated.

4 new tests covering both error paths and both search annotation cases.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard tool/skills Skills system (list, view, manage) labels May 2, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for improving the skills-hub failure message. The generic failure remains on current main at hermes_cli/skills_hub.py:545, so the underlying UX problem is still valid.

Problems

  • The new meta is not None branch at hermes_cli/skills_hub.py:343 is not sufficient proof of a GitHub 404. On current main, HermesIndexSource.inspect() returns index metadata without fetching (tools/skills_hub.py:3886-3892), while GitHubSource.fetch() maps all download failures to None (tools/skills_hub.py:592-606). This can call a rate-limit or transport failure a stale entry.
  • Current do_install() retains a rate-limit hint at hermes_cli/skills_hub.py:539-553 (added by 7e0e5ea03). The PR branch predates that path, so the stale branch needs to preserve it.

Suggested changes

  • Carry a confirmed fetch status/reason through resolution, and reserve the stale-index message for a matching confirmed 404.
  • Add 404, rate-limit, and generic-fetch-failure cases; the current fake source at tests/hermes_cli/test_skills_hub.py:258 returns bare None, so it cannot validate the proposed distinction.

Automated hermes-sweeper review.

Comment thread hermes_cli/skills_hub.py

if not bundle:
c.print(f"[bold red]Error:[/] Could not fetch '{identifier}' from any source.\n")
if meta is not None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meta only proves that some source returned metadata. On current main the centralized index returns metadata without fetching, while GitHub fetch reduces 404s, rate limits, and other failures to None; please gate this message on a confirmed matching 404 and retain the existing rate-limit hint.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 12, 2026
teknium1 pushed a commit that referenced this pull request Sep 12, 2026
_resolve_source_meta_and_bundle already distinguishes index-hit-without-
files from unknown identifiers, but do_install printed the same generic
'Could not fetch' for both, sending users off to re-check spellings for
what is actually a stale skills.sh entry. Split the message, and add a
staleness caveat to do_search results from skills.sh.

Fixes #3259. Supersedes #3261 (stale since July — re-applied onto the
current _print_fetch_failure helper).
teknium1 added a commit that referenced this pull request Sep 12, 2026
…p per-search caveat

A throttled GitHub fetch also yields index-metadata-without-bundle, so the new
stale-entry verdict would tell users a skill "no longer exists upstream" when
it does. Check the adapters' rate-limit flag first and keep the existing
rate-limit hint for that case (the keep_open review concern on #3261).

The per-search "results may be stale" note is dropped: it fires on every
skills.sh search whether or not anything is stale, and the install-time error
now names the condition precisely where it happens.
@teknium1

Copy link
Copy Markdown
Collaborator

Closing — superseded by #108906 (257a704d18d1 from @nikkoxgonzales' #106901 + 08bb58bd4a58). You were the earliest to propose naming the stale index entry; the landed version follows the same design and adds the guard the review asked for here (don't mislabel a rate-limited fetch as stale). Thanks; #3259 is closed.

@teknium1 teknium1 closed this Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/skills Skills system (list, view, manage) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: skills.sh index contains non-existent skills - install fails with "Could not fetch"

3 participants