fix(skills): centralized index + tree cache — eliminate rate-limit failures on install - #8575
Merged
Conversation
…n install
Skills.sh installs hit the GitHub API 45 times per install because the
same repo tree was fetched 6 times redundantly. Combined with search
(23 API calls), this totals 68 — exceeding the unauthenticated rate
limit of 60 req/hr, causing 'Could not fetch' errors for users without
a GITHUB_TOKEN.
Changes:
- Add _get_repo_tree() cache to GitHubSource — repo info + recursive
tree fetched once per repo per source instance, eliminating 10
redundant API calls (6 tree + 4 candidate 404s)
- _download_directory_via_tree returns {} (not None) when cached tree
shows path doesn't exist, skipping unnecessary Contents API fallback
- _check_rate_limit_response() detects exhausted quota and sets
is_rate_limited flag
- do_install() shows actionable hint when rate limited: set
GITHUB_TOKEN or install gh CLI
Before: 45 API calls per install (68 total with search)
After: 31 API calls per install (54 total with search — under 60/hr)
Reported by community user from Vietnam (no GitHub auth configured).
…or search/install Add a CI-built skills index served from the docs site. The index is crawled daily by GitHub Actions, resolves all GitHub paths upfront, and is cached locally by the client. When the index is available: - Search uses the cached index (0 GitHub API calls, was 23+) - Install uses resolved paths from index (6 API calls for file downloads only, was 31-45 for discovery + downloads) Total: 68 → 6 GitHub API calls for a typical search + install flow. Unauthenticated users (60 req/hr) can now search and install without hitting rate limits. Components: - scripts/build_skills_index.py: Crawl all sources (skills.sh, GitHub taps, official, clawhub, lobehub), batch-resolve GitHub paths via tree API, output JSON index - tools/skills_hub.py: HermesIndexSource class — search/fetch/inspect backed by the index, with lazy GitHubSource for file downloads - parallel_search_sources() skips external API sources when index is available (0 GitHub calls for search) - .github/workflows/skills-index.yml: twice-daily CI build + deploy - .github/workflows/deploy-site.yml: also builds index during docs deploy Graceful degradation: when the index is unavailable (first run, network down, stale), all methods return empty/None and downstream sources handle the request via direct API as before.
teknium1
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A community user in Vietnam (no GitHub auth) searched for a skill then tried to install it. The search consumed 23 GitHub API calls (scanning default taps) and the install needed 45 more (the same repo tree fetched 6 times redundantly). Total: 68 calls — exceeding the unauthenticated rate limit of 60/hr, causing
Could not fetchwith no explanation.Solution: Two-layer fix
Layer 1: Tree cache (immediate — helps all users now)
GitHubSource._get_repo_tree()caches repo info + recursive tree per-instanceLayer 2: Centralized skills index (long-term — eliminates the problem)
hermes-agent.nousresearch.com/docs/api/skills-index.json)scripts/build_skills_index.pycrawls all sources twice daily, batch-resolves GitHub paths via tree APIHermesIndexSourceintools/skills_hub.py— search/fetch/inspect backed by cached indexparallel_search_sources()skips external API sources entirelyImpact
Graceful degradation: when the index is unavailable (first run, network issue), the tree-cached direct API path handles it (54 calls, under limit).
Files changed
tools/skills_hub.pyHermesIndexSource, search optimizationhermes_cli/skills_hub.pyscripts/build_skills_index.py.github/workflows/skills-index.yml.github/workflows/deploy-site.yml.gitignoreTest plan