Skip to content

fix(prompt_builder): include skills manifest digest in cache key - #18443

Open
leon7609 wants to merge 1 commit into
NousResearch:mainfrom
leon7609:fix/prompt-builder-skills-cache-invalidation
Open

fix(prompt_builder): include skills manifest digest in cache key#18443
leon7609 wants to merge 1 commit into
NousResearch:mainfrom
leon7609:fix/prompt-builder-skills-cache-invalidation

Conversation

@leon7609

@leon7609 leon7609 commented May 1, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a stale-cache bug in build_skills_system_prompt's in-process LRU.

The cache key was based only on directory paths (skills_dir.resolve() + a tuple of external dir paths). Edits to skill files within those directories — add, edit, remove, rename — did not invalidate the cache, so a long-running agent kept serving a stale system prompt until the process restarted.

The disk snapshot (Layer 2) already validates against a manifest of mtime/size pairs; this PR folds the same manifest digest into the in-process cache key (Layer 1) so both layers share invalidation semantics. Two new helpers — _skills_manifest_digest() and _skills_cache_state() — compute a per-directory SHA-1 over a sorted manifest and feed the result into the existing cache key tuple.

Related Issue

No existing issue. Discovered while iterating on a skill — agent kept serving the old prompt until I restarted the gateway.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • agent/prompt_builder.py
    • Add _skills_manifest_digest(skills_dir) — SHA-1 over a sorted JSON dump of _build_skills_manifest()'s output.
    • Add _skills_cache_state(skills_dir, external_dirs) — tuple of (resolved_path, digest) pairs across local + external dirs.
    • Replace tuple(str(d) for d in external_dirs) in the cache key with _skills_cache_state(...).
  • tests/agent/test_prompt_builder.py — extend with cases that mutate skill files and assert the cache key changes.

How to Test

  1. Start a session with skills loaded: hermes chat -q "list skills".
  2. Edit any SKILL.md content in ~/.hermes/skills/<some-skill>/ (without restarting hermes).
  3. Trigger another prompt-build (e.g., new chat session in the same process).
  4. Before this fix: the in-process cache returns the old skills prompt — agent doesn't see the edit.
  5. After this fix: the manifest digest differs, cache misses, and the new content is loaded.

Or run the unit tests directly:

pytest tests/agent/test_prompt_builder.py -v

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(prompt_builder): ...)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run pytest tests/ -q and the changes don't introduce new failures (vs. main baseline)
  • I've added tests for the change
  • I've tested on my platform: macOS 15 (Apple Silicon)

Documentation & Housekeeping

  • No public API surface changed
  • No config schema changed

The in-process LRU cache for `build_skills_system_prompt` was keyed
only on directory paths (`skills_dir.resolve()` plus a tuple of
external dir paths).  Edits to skill files within those directories
— add, edit, remove, rename — did not invalidate the cache, so a
running agent kept serving a stale system prompt until the process
restarted.

The disk snapshot (Layer 2) already validates against a manifest of
mtime/size pairs; this commit folds the same manifest digest into
the in-process cache key (Layer 1) so both layers share invalidation
semantics.

Add `_skills_manifest_digest()` and `_skills_cache_state()` helpers
and use the latter in place of the bare external-dir tuple.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating a real Layer-1 cache staleness path. Current main still keys the skills LRU by directory paths at agent/prompt_builder.py:1481-1489 and returns it before Layer-2 manifest validation at :1490-1497.

Problems

  • The new _skills_cache_state() calls _build_skills_manifest() before every LRU lookup. On current main, that function performs os.walk() and os.stat() across indexed files (agent/prompt_builder.py:1281-1297), so unchanged cache hits regain the filesystem traversal that cache commit 5127567d5 specifically reduced from 546ms to <1ms.
  • Current reload behavior deliberately preserves this prompt cache: agent/skill_commands.py:405-416 rescans commands and queues a user-turn note instead of invalidating the system-prompt cache. The intended automatic-refresh boundary needs a maintainer decision before this changes it implicitly.
  • The test adds only a local-file-addition case; it does not cover edits, removals, renames, external directories, or unchanged-hit performance.

Suggested changes

  • Establish the intended refresh boundary, then preserve a cheap unchanged LRU hit.
  • Add coverage for each claimed mutation class and external directories.

Automated hermes-sweeper review.

Comment thread agent/prompt_builder.py


def _skills_manifest_digest(skills_dir: Path) -> str:
"""Return a stable digest of one skills directory's manifest."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This manifest is evaluated before the Layer-1 lookup, so every otherwise-unchanged cache hit now walks and stats the complete skills tree. Current _build_skills_manifest() uses os.walk() plus per-file os.stat(); please retain a cheap unchanged-hit path before adding this to the key.

@teknium1 teknium1 added sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) tool/skills Skills system (list, view, manage) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants