feat(context-files): add @<path> include expansion (CAAMP-style transitive includes)

[kryptobaseddev]

Summary

Adds first-class @<path> include expansion to all context files (AGENTS.md, .hermes.md, CLAUDE.md, .cursorrules, .cursor/rules/*.mdc, SOUL.md). This brings Hermes to parity with Claude Code, Cursor, and other harnesses that already honor @-includes, and unlocks the "single source of truth" pattern where one canonical instruction file lives in ~/.agents/AGENTS.md (or anywhere) and is referenced from every project.

Before this change, a project AGENTS.md containing:

@~/.agents/AGENTS.md

# Project rules below
...

…was injected into the system prompt with the literal string @~/.agents/AGENTS.md instead of the global file's contents. Users had to copy/symlink shared rules into every repo, or accept that their CAAMP-style includes only worked in Claude Code, not Hermes.

Now the expander recursively resolves the include, follows transitive includes (the global file can reference its own SSoT files), and produces a fully-assembled context block — with depth caps, cycle detection, and prompt-injection scanning applied to every layer.

Design

Implementation is split into a pure, reusable module (agent/context_includes.py) and a thin wiring layer in agent/prompt_builder.py. The expander module has zero dependencies on prompt-builder internals — the prompt-injection scanner and size-truncator are passed in as hooks. This means cron jobs, gateway hooks, batch_runner, or third-party plugins can reuse the same expander without dragging in the full prompt-builder graph.

Public API (agent/context_includes.py)

expand_includes(
    content: str,
    base_dir: Path,
    *,
    depth: int = 0,
    visited: set[str] | None = None,
    scanner: Callable = identity,
    truncator: Callable = identity,
    max_depth: int = CONTEXT_INCLUDE_MAX_DEPTH,  # default 5
) -> str

Returns the expanded content with each @<path> directive replaced by the wrapped contents of the target file.

Loading pipeline (per context file)

read_text -> expand_includes -> _scan_context_content -> _truncate_content

The scanner runs against the expanded text so prompt-injection patterns hidden in an included file are still blocked. The truncator runs both per-include (inside expand_includes) and over the final assembled context, preserving the existing 20K char-per-source budget.

Syntax

A line that contains only @<path> (with optional leading/trailing whitespace) is replaced inline with the contents of the referenced file. Inline @mentions in prose (e.g. "ask @bob") are not expanded.

Form	Resolves against
@/abs/path.md	absolute path as-is
@~/path.md	$HOME
@$VAR/path.md or @${VAR}/path.md	environment variable
@relative/path.md	the including file's directory (not CWD)

Safety rails

Guard	Behavior	Marker emitted
Max depth (default 5)	Includes deeper than max_depth are stopped	<!-- @max-depth: <path> -->
Cycle detection	A → B → A breaks the loop via visited-set	<!-- @cycle: already-included <path> -->
Missing file	Surrounding content preserved	<!-- @missing: <path> ... -->
Unreadable file	Surrounding content preserved	<!-- @unreadable: <path> (<err>) -->
Per-include size cap	Same 20K head/tail truncation as top-level files	standard truncation marker
Prompt-injection scan	Same patterns as top-level context files	[BLOCKED: <path> contained ...]
Code fences	@<path> inside ``` or ~~~ blocks stays as literal text	(no expansion)

Successfully-expanded chunks are wrapped with begin/end markers so users (and the agent) can see exactly which file each piece came from:

<!-- @include-begin: ~/.agents/AGENTS.md -->
...included content...
<!-- @include-end: ~/.agents/AGENTS.md -->

Tests

33 new tests, all passing, zero regressions in the existing suite.

• tests/agent/test_context_includes.py — 22 unit tests for the standalone module:

  • resolve_include_path: absolute, relative, ~, $VAR
  • INCLUDE_PATTERN: line-only matching, leading whitespace, inline rejection
  • expand_includes: simple, nested A→B→C, cycles, missing files, max-depth marker, custom max-depth, code fences (``` and ~~~), scanner/truncator hook invocation, relative-path resolution inside included files, directory-target handling, marker presence

• tests/agent/test_prompt_builder.py::TestExpandIncludes — 11 integration tests verifying the wiring through the full build_context_files_prompt pipeline.

$ .venv/bin/python -m pytest tests/agent/test_context_includes.py tests/agent/test_prompt_builder.py -q
149 passed in 5.78s

Full tests/agent/ suite: 2385 passing (5 pre-existing failures from missing optional deps botocore/fastapi on main, unrelated to this PR).

End-to-end verification

Tested against a real project that uses the SSoT pattern (project AGENTS.md → @~/.agents/AGENTS.md → @~/.cleo/templates/CLEO-INJECTION.md):

from agent.prompt_builder import build_context_files_prompt
out = build_context_files_prompt(cwd='/path/to/project')
# expanded global include:        True  ✓
# expanded transitive include:    True  ✓  (nested SSoT works)
# included content present:       True  ✓
# no raw @-include leak:          True  ✓

Backward compatibility

100% backward compatible. Files without @-include directives behave exactly as before (the regex pass is a no-op). Existing tests untouched.

Files changed

 agent/context_includes.py                         | 250 +++++++++++++++++ (new)
 agent/prompt_builder.py                           |  30 ++
 tests/agent/test_context_includes.py              | 207 ++++++++++++++ (new)
 tests/agent/test_prompt_builder.py                | 144 ++++++++++
 website/docs/user-guide/features/context-files.md |  85 ++++++
 5 files changed, 716 insertions(+)

Commits

• feat(context-includes): add reusable @-include expander module — pure module + 22 tests
• feat(prompt-builder): expand @<path> includes in all context files — wiring + 11 tests
• docs(context-files): document @-include syntax, safety rails, and SSoT pattern — user-facing docs

Out of scope (potential follow-ups)

• Config flag to disable include expansion (context_files.expand_includes: false)
• Glob support (@~/.agents/skills/*.md) — current pattern is one file per directive
• @!<path> opt-out from injection scanning — probably never wanted

[teknium1]

Thanks for the well-tested include-expansion implementation. The underlying gap is still present on current main: agent/prompt_builder.py:1882-1885 reads and scans literal AGENTS.md content without expansion.

Problems

• agent/context_includes.py:112-117 accepts ~, environment-expanded, and absolute paths, then reads the result into the model context. This conflicts with the containment rationale in agent/subdirectory_hints.py:172-195, which prevents loading context outside the workspace.
• Current main’s truncation contract now carries context_length and read_path (agent/prompt_builder.py:1779-1816); the PR’s older wrapper does not preserve those per-source semantics for included files.
• The diff also changes the compatibility shim docker/entrypoint.sh from executable to mode 100644; current docs still support it for downstream entrypoint overrides (website/docs/user-guide/docker.md:499-500).

Suggested changes

• Establish an explicit trusted-root or user-approved mechanism for external includes, preserve subdirectory containment, and test external-path rejection.
• Integrate with the current context-budget/truncation API and remove unrelated Docker changes.

Automated hermes-sweeper review.

[teknium1]

This lets a repository-controlled context file resolve ~, environment variables, and arbitrary absolute paths before the expander reads the target into the model prompt. That bypasses the existing workspace-containment policy in agent/subdirectory_hints.py:172-195; please restrict this to an explicit trusted root or user-approved external source and add a rejection test.