feat(agent): universal instruction file discovery with 50K budget

[tjp2021]

Problem

Hermes only reads its own instruction files (.hermes.md, AGENTS.md, CLAUDE.md, .cursorrules). Users switching from other AI coding agents lose their project instructions entirely — they have to manually recreate them in a Hermes-compatible format. This creates unnecessary adoption friction.

Issue: #524

Solution

Replace first-match-wins with load-all-that-exist within a 50K character global budget. Hermes-native files get priority; third-party files fill remaining budget.

Tiered Priority System

Tier	Files	Discovery
0 (highest)	.hermes.md / HERMES.md	Walks to git root
1	AGENTS.md, CLAUDE.md, .cursorrules	CWD
2	.github/copilot-instructions.md, GEMINI.md, codex.md, .clinerules, .roorules, .windsurfrules, .augment-guidelines, .goose/instructions.md, .goosehints, .tabnine, .sourcegraph/instructions.md	CWD
Dirs	.cursor/rules/*.mdc, .cline/rules/*.md, .roo/rules/*.md, .windsurf/rules/*.md, .amazonq/rules/*.md, .kiro/steering/*.md, .gemini/*.md	CWD

Key Details

• 50K char global budget — files load in priority order until budget is exhausted
• Case-insensitive deduplication — AGENTS.md and agents.md won't double-load on case-insensitive filesystems
• SOUL.md exempt from budget (loaded separately as agent identity)
• 3 new prompt injection patterns — XML role injection (<|im_start|system>), roleplay injection ("you are now..."), delimiter injection (---\nsystem:)
• Backward compatible — existing .hermes.md / AGENTS.md / CLAUDE.md / .cursorrules behavior unchanged; they just aren't exclusive anymore
• Security edge cases tested — binary file handling (.tabnine with raw bytes), symlink behavior documented

Why This Matters

Users already have instruction files from their current tools — .github/copilot-instructions.md, .cursorrules, .clinerules, etc. This change means those files work in Hermes without any migration step. Users can try Hermes in an existing project and their conventions carry over automatically.

Files Changed

File	Change
agent/prompt_builder.py	Replace 3 single-file loaders with generic _load_instruction_file() / _load_instruction_dir(), add file registries, budget tracking, dedup, 3 new threat patterns
tests/agent/test_prompt_builder.py	TestUniversalContextDiscovery class — parametrized tests for all 11 tier-2 files, all 7 rule dirs, multi-file loading, budget cap, dedup, injection blocking, binary files, symlinks, backward compat
website/docs/developer-guide/prompt-assembly.md	Updated architecture docs
website/docs/user-guide/features/context-files.md	Updated user-facing docs
website/docs/user-guide/configuration.md	Updated config reference

Test plan

• pytest tests/agent/test_prompt_builder.py — 149 passed, 2 skipped
• All existing tests unchanged (backward compatible)
• Parametrized coverage for every new file type and directory
• Budget enforcement: 50K cap stops loading when exhausted
• Case-insensitive dedup: GEMINI.md + gemini.md loads once
• Injection blocking: all 3 new patterns tested
• Binary file handling: raw bytes in .tabnine silently skipped
• Symlink behavior: documented and tested (follows symlinks, consistent with existing code)
• Manual: Created temp dir with .cursorrules + GEMINI.md + .clinerules, called build_context_files_prompt(), verified all three files present in output with correct headers and content

🤖 Generated with Claude Code

[teknium1]

Thanks for the broad compatibility work. Universal instruction-file discovery is still absent on current main: agent/prompt_builder.py:1976-1981 remains a first-match-wins chain.

Problems

• The advertised 50K global cap is not enforced. The added loop appends a full section and only then subtracts its size (agent/prompt_builder.py:842). With the PR test's three 18K tier-1 files (tests/agent/test_prompt_builder.py:1095-1104), the assembled project context exceeds 50K; the test only confirms that a later file is skipped.
• Current main now passes context_length into context loading (agent/system_prompt.py:451-453) and derives configurable/dynamic caps (agent/prompt_builder.py:1201-1219). The salvage must preserve that interface.
• Main uses the shared scanner imported at agent/prompt_builder.py:47; additional threat signatures need to land in tools/threat_patterns.py, not restore the old local registry.

Suggested changes

• Enforce remaining aggregate budget before append and assert the resulting project-context size is capped (excluding SOUL.md).
• Port the loaders to current dynamic/configured cap semantics.
• Route any new scanner patterns through the shared threat-pattern library and its tests.

Automated hermes-sweeper review.

[teknium1]

This only checks budget_remaining before loading the next source, so a source larger than the remaining allowance is appended whole and can take total project context beyond 50K. Truncate or reject section against the remaining budget before sections.append, and add an assertion on the final aggregate size.

[GottZ]

This was generated by AI during triage.

Summary

Four PRs touch context-file prompt assembly, but they address three distinct causes: #4098 replaces first-match discovery with universal loading, #23057/#23062 remove the obsolete SOUL.md fallback path, and #23331 adds cwd-independent HERMES_HOME/AGENTS.md policy loading. They should therefore not be treated as four interchangeable implementations of one fix.

Related pull requests

• #4098 related — (+372/-103) — keep open for salvage, not merge as-is: the diff adds broad third-party instruction discovery and load-all behavior, but the contributor review identifies an unenforced 50K aggregate cap, a stale context_length interface, and threat patterns placed in the obsolete local registry; those defects directly undermine the advertised budget and current-main compatibility.
• #23057 [closed] related — (+29/-51) — closed but still relevant as the earlier identical implementation: the diff removes skip_soul and keeps SOUL.md exclusively in the identity layer. Its recorded state establishes only that it is closed; no evidence here establishes why it was closed or that #23062 formally superseded it.
• #23062 related — (+29/-51) — keep open only if ported to current main: although its diff correctly targets the redundant SOUL.md context path, the contributor keep_open review notes that it edits the obsolete run_agent.py call site, would leave agent/system_prompt.py passing the removed argument, and omits the matching Chinese documentation update.
• #23331 related — (+266/-12) — strongest merge candidate after exact-head verification: the diff directly fixes missing global operational policy by loading HERMES_HOME/AGENTS.md independently of cwd, and its updated implementation adds the contributor-requested cwd-equals-HERMES_HOME deduplication while preserving HERMES.md precedence and preventing CLAUDE.md fall-through. Despite the keep_open review on #23331, the shown diff now addresses both of its concrete blockers, and subsequent production feedback reports the duplication fixed with 176/176 tests passing; the documented skip_context_files exclusions remain intentional scope limits rather than claimed coverage.

Duplicates

#23057 and #23062 have effectively identical diffs. #23057 is a closed earlier instance, but the supplied evidence does not establish its closure reason or a formal supersession relationship.

Suggested consolidation

Merge #23331 after exact-head CI confirms the updated deduplication tests and documentation; it is the only shown PR that directly and currently addresses HERMES_HOME/AGENTS.md policy loss. Keep #4098 separate for its broader universal-discovery work and required budget/current-main repairs, and keep #23062 separate only if the SOUL.md cleanup is still desired and is ported to agent/system_prompt.py with translated docs; #23057 is already closed and should be referenced only as the effectively identical earlier diff, not described as formally superseded.

Cross-PR triage: Reviewed 4 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 71 kB of PR diffs, 10 kB of issue/PR text, 11 kB of discussion (8 comments), 1 verify verdict. verdicts reflect diff content, not PR titles. Part of an automated triage batch.