Add opt-in Headroom Phase 1 tool-output compression plugin

[channkim]

Summary

Promote the Headroom structured tool-output compression experiment to Phase 1 as disabled-by-default code + tests only.

• Add bundled opt-in headroom plugin using the existing transform_tool_result hook.
• Add default-disabled headroom config block.
• Hard-limit Phase 1 to search_files and browser_snapshot; config can narrow but not widen the allowlist.
• Keep explicit exclusions for terminal, read_file, delegate_task, patch, write_file, memory, send_message, clarify, and cronjob.
• Add kill switches via config/env.
• Do not introduce raw-result storage or retrieval handles in Phase 1; compressed payloads mark retrieval unavailable.
• Redact secret-like strings and dict keys before constructing compressed payloads.
• Preserve existing untrusted wrapper prefix/suffix when compressing wrapped results.
• Handle real search_files JSON plus pagination hint suffix.

Non-goals / invariants

• No default live behavior change: plugin loading alone remains identity because headroom.enabled defaults to false.
• No default-profile canary/live rollout in this PR.
• No raw output persistence in this PR.
• No expansion beyond the Phase 1 allowlist.
• Mandatory excluded tools cannot be re-enabled by config.

Validation

Ran locally:

scripts/run_tests.sh tests/plugins/test_headroom_plugin.py
scripts/run_tests.sh tests/plugins/test_headroom_plugin.py tests/test_transform_tool_result_hook.py tests/hermes_cli/test_plugin_scanner_recursion.py tests/hermes_cli/test_plugins.py
python -m py_compile plugins/headroom/__init__.py tests/plugins/test_headroom_plugin.py
git diff --check

Results:

• tests/plugins/test_headroom_plugin.py: 19 passed
• combined focused suite: 116 passed
• py_compile: passed
• git diff --check: passed

Review

Codex adversarial review initially found two blockers:

1. headroom.excluded_tools was exposed but not read.
2. real truncated search_files output appends a plaintext hint after JSON, causing parsing to skip compression.

Both were fixed and re-reviewed. Final Codex review verdict: PASS.

Handoff

• Branch: feat/headroom-phase1
• Commit: 8e5acc7b5
• Implementation files:
  • plugins/headroom/__init__.py
  • plugins/headroom/plugin.yaml
  • hermes_cli/config.py
  • tests/plugins/test_headroom_plugin.py

Follow-ups before Phase 2

• Storage/retrieval handle design review if raw output persistence is introduced.
• Verify /compress, resume, missing handle, kill switch, scope isolation, and untrusted wrapper tests before any live canary.
• Phase 2 must be in a separate test profile only, not default profile.

[teknium1]

Thanks for keeping the experiment opt-in and using the existing transform_tool_result seam.

Problems

• Current main changed the search_files wire shape in merged #47866: tools/file_tools.py:2004 requests densify=True, and tools/file_operations.py:281-297 emits matches_text for five or more matches. This PR's _compress_search_files() only recognizes matches, files, and counts, so it returns no transformation for the current densified result. The added tests exercise only the older matches array.
• The PR adds non-secret HERMES_HEADROOM_* enablement, kill-switch, and allowlist settings. website/docs/user-guide/configuration.md:53-60 requires behavioral settings to live in config.yaml.

Suggested changes

• Re-scope search_files around the native lossless representation from #47866, and add real-path coverage for matches_text plus the below-threshold array form.
• Keep activation and limits in config.yaml; remove the new behavioral environment overrides.

Automated hermes-sweeper review.

[teknium1]

Current main's search_files output uses matches_text rather than matches when it has at least five matches (tools/file_tools.py:2004, tools/file_operations.py:281-297). This branch will therefore skip the common current result shape; add coverage using the real densified payload or re-scope this transformer around the native lossless representation.