Skip to content

perf(skills): list categories in system prompt - #26842

Closed
haran2001 wants to merge 1 commit into
NousResearch:mainfrom
haran2001:perf/two-level-skill-index
Closed

haran2001 wants to merge 1 commit into
NousResearch:mainfrom
haran2001:perf/two-level-skill-index

Conversation

@haran2001

Copy link
Copy Markdown
Contributor

Summary

  • Replace the full per-skill system-prompt index with a category-level index and skill counts.
  • Instruct agents to call skills_list(category=...) before loading matching skills with skill_view(...).
  • Update prompt-builder tests for category-only rendering, filtering, and conditional skill visibility.

Test Plan

  • python -m pytest tests/agent/test_prompt_builder.py -q -o 'addopts='

Closes #26786

@alt-glitch alt-glitch added type/perf Performance improvement or optimization P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/skills Skills system (list, view, manage) labels May 16, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting a meaningful fixed prompt-cost source. The full-index premise still exists on current main (agent/prompt_builder.py:1660-1667), but the proposed global category-only strategy conflicts with a later verified design decision.

Problems

  • Current main intentionally keeps every skill name visible: agent/prompt_builder.py:1622-1628 and commit ee1a744ac document that models did not reliably use skills_list to rediscover omitted agent-created skills. The PR's skills_list(category=...) replacement at agent/prompt_builder.py:1177-1179 reintroduces that failure mode.
  • Renaming the block at agent/prompt_builder.py:1199-1201 leaves agent/context_breakdown.py:15 and hermes_cli/prompt_size.py:21 unable to find and attribute the skills index.

Suggested changes

  • Preserve name visibility and reduce description noise only, following the current names-only demotion contract and tests at tests/agent/test_prompt_builder.py:429-475.
  • Retain the existing wrapper tag or update all block consumers and their tests.

This is an automated hermes-sweeper review.

Comment thread agent/prompt_builder.py
"Before replying, scan the skills below. If a skill matches or is even partially relevant "
"to your task, you MUST load it with skill_view(name) and follow its instructions. "
"Before replying, scan the skill categories below. If a category matches or is even partially relevant "
"to your task, you MUST call skills_list(category=...) to inspect the available skills, then load "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main intentionally does not rely on skills_list to rediscover omitted skills: ee1a744ac records a real failure where an agent-created skill disappeared from the index and was not rediscovered. Preserve skill names rather than making this category-only.

Comment thread agent/prompt_builder.py
"pitfalls you discovered, update it before finishing.\n"
"\n"
"<available_skills>\n"
"<available_skill_categories>\n"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This renamed wrapper is not consumed by the current prompt diagnostics: agent/context_breakdown.py:15 and hermes_cli/prompt_size.py:21 only match <available_skills>. Update those consumers and tests, or retain the existing tag.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) labels Jul 13, 2026
@kshitijk4poor

Copy link
Copy Markdown
Contributor

Closing on the design axis after checking the current skills-index implementation: main deliberately keeps every skill visible in the index at all times — the never-hide decision (ee1a744) demotes long categories to names-only under pressure but never removes entries, precisely because models do NOT reliably reach for skills_list to rediscover what the index stops showing them. This PR's categories-only index (with a 'call skills_list(category=...) to inspect' nudge) is the exact pattern that decision rejected: in practice the model skips the extra round-trip and misses skills. The category-count suffix rider is neutral-to-negative once full entries stay (extra tokens, no signal). Thank you — the token concern is real, and the compact-guidance angle (#72813) is the shape of it we can take.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/skills Skills system (list, view, manage) type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: two-level skill index in system prompt to reduce fixed token overhead

4 participants