Skip to content

fix(skills-index): give the offline builder a walk budget the full ClawHub catalog fits in - #44494

Closed
AIalliAI wants to merge 1 commit into
NousResearch:mainfrom
AIalliAI:fix/clawhub-index-walk-budget
Closed

fix(skills-index): give the offline builder a walk budget the full ClawHub catalog fits in#44494
AIalliAI wants to merge 1 commit into
NousResearch:mainfrom
AIalliAI:fix/clawhub-index-walk-budget

Conversation

@AIalliAI

Copy link
Copy Markdown
Contributor

What does this PR do?

Unblocks the skills index / docs-site deploys, which have failed on every run since 2026-06-10 06:22 UTC.

Root cause: the interactive 12-second CATALOG_WALK_BUDGET_SECONDS added in 105625d ("honour overall_timeout and bound ClawHub catalog walk", the #38459 hang fix) also applies to the offline index builder. The builder's walk-to-exhaustion over the ~50k-skill ClawHub catalog needs ~250 sequential pages (~260s at the observed ~1 page/s), so the 12s budget truncates it at ~16 pages. The EXPECTED_FLOORS health check from #42347 then correctly refuses to ship the degenerate index (clawhub: 1797 < expected floor 200003195 < 20000), failing skills-index.yml and deploy-site.yml ever since — the live docs site is frozen at the June 9 state and the watchdog keeps bumping #38240.

Evidence

  • Last green index run: 2026-06-09 18:52 UTC; first red: 2026-06-10 07:41 UTC — the budget commit landed on main 2026-06-10 06:22 UTC, in between.
  • First failing run logs clawhub: 1797 skills (12.9s) — the crawl stops at exactly the 12s budget; later runs report 3195, i.e. the count varies with network speed, which is a truncation signature, not a catalog shrink.
  • Live repro: a 30s budget walks 5,788 skills; 12s walks ~3.2k — linear in the budget.

Fix: ClawHubSource now accepts an explicit per-instance catalog_walk_budget_seconds; scripts/build_skills_index.py passes 600s (≈2.3× the measured full-walk time). Interactive callers (browse/search cold start) keep the tight 12s default, so the #38459 hang fix is untouched.

Related Issue

Fixes #38240

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/skills_hub.pyClawHubSource.__init__(catalog_walk_budget_seconds=None) instance override of the class default
  • scripts/build_skills_index.py — builder constructs ClawHubSource(catalog_walk_budget_seconds=600)
  • tests/tools/test_skills_hub_clawhub.py — regression test: instance budget wins over an expired class-level deadline, walk completes and caches
  • tests/scripts/test_build_skills_index_health.py — fake-source stub accepts the new kwarg

How to Test

  1. pytest tests/tools/test_skills_hub_clawhub.py tests/scripts/test_build_skills_index_health.py -q → 18 passed
  2. pytest tests/tools -k skills_hub -q → 166 passed, 1 skipped
  3. Live: ClawHubSource(catalog_walk_budget_seconds=30)._load_catalog_index() gathers ~5.8k skills vs ~3.2k at the 12s default; with 600s the walk reaches cursor exhaustion (~50k) and the health check floor passes
  4. After merge, gh workflow run skills-index.yml should go green and the watchdog probe on [skills-index-watchdog] Skills index is stale or degraded (degraded) #38240 should recover

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run the relevant tests and all pass (see How to Test)
  • I've added tests for my changes
  • I've tested on my platform: macOS

Documentation & Housekeeping

  • I've updated relevant documentation — N/A (internal CI/index builder behavior)
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact — N/A (no platform-specific code)
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

First failing run (skills-index #27261057718):

  clawhub: 1797 skills (12.9s)
ERROR: skills index health check failed — refusing to ship a degenerate index.
  clawhub: 1797 < expected floor 20000

Latest deploy-site failure (#27366487914): clawhub: 3195 < expected floor 20000

…awHub catalog fits in

The interactive 12s CATALOG_WALK_BUDGET_SECONDS (added to stop browse/
search hanging on a slow catalog walk) also bounds the offline index
builder, truncating the ~50k-skill ClawHub walk at ~16 pages. Every
skills-index/deploy-site run since 2026-06-10 06:22 UTC fails the
EXPECTED_FLOORS health check (clawhub 1797-3195 < 20000), freezing the
docs site and skills index.

Let ClawHubSource take an explicit per-instance walk budget and pass
600s from scripts/build_skills_index.py (full walk needs ~260s at the
observed ~1 page/s). Interactive callers keep the 12s default.

Fixes NousResearch#38240
@alt-glitch alt-glitch added type/bug Something isn't working tool/skills Skills system (list, view, manage) P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jun 11, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #44500 (merged) — that PR already unblocked the skills index by letting the offline builder walk the full ClawHub catalog (unbounded max_items=0), fixing #38240. This change targets the same root cause via a catalog_walk_budget_seconds=600 override; the fix is already on main.

@AIalliAI AIalliAI closed this Jun 12, 2026
AIalliAI added a commit to AIalliAI/Hermes that referenced this pull request Jun 14, 2026
…ousResearch#44500)

Upstream implementation by @austinpickett, merged to main as 2ee69d0.
Replaces our reverted local variant (2621a96, PR NousResearch#44494 closed as
superseded): instead of an explicit 600s builder budget, the wall-clock
budget now applies only to bounded browse walks (max_items > 0) and the
index builder uses the unbounded path via SOURCE_LIMITS clawhub=0.

(patch applied from PR NousResearch#44500 diff; shallow clone prevented a direct
cherry-pick of 2ee69d0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists tool/skills Skills system (list, view, manage) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[skills-index-watchdog] Skills index is stale or degraded (degraded)

3 participants