fix: redact secrets in non-code files (.env, .yaml, .json)

[zhangjinzan1]

Problem

redact_sensitive_text is called with code_file=True for ALL files in both read_file and search_files tools (see tools/file_tools.py lines 620 and 1045). This skips the ENV-assignment and JSON-field regex patterns globally, which means secrets in config files like .env, .yaml, .toml, and .json are not properly redacted.

For example, WEIXIN_TOKEN=abcdef123456 in a .env file would only be partially masked (by prefix patterns) instead of fully redacted by the ENV-assignment pattern.

Fix

Introduce _is_source_code_file(path) that checks the file extension against a known set of source-code extensions (.py, .js, .go, .rs, .ts, .java, etc.). Only source-code files use code_file=True; config and env files use code_file=False so that ENV-assignment and JSON-field patterns are applied.

This preserves the original intent (avoid false positives in source code like MAX_TOKENS=100) while properly protecting secrets in config files.

Testing

• All existing test_file_tools.py tests pass (29/29)
• All redact-related tests pass (155 passed, 3 skipped)
• Verified manually: .env file contents are now fully redacted, .py files still skip ENV patterns

Before/After

File type	Before (code_file=True)	After (code_file=False)
.env WEIXIN_TOKEN=abcdef...	Partial mask abcdef...7890	Full mask ***
.env XIAOMI_API_KEY=sk-...	Prefix pattern mask	ENV pattern full mask
.py MAX_TOKENS=100	Skipped (no false positive)	Skipped (unchanged)

[tonydwb]

Code Review Summary: PR #35246

Verdict: APPROVE ✅

Overall: Precise, well-scoped fix with excellent PR description (before/after table, rationale, edge case documentation).

✅ Looks Good

• Correct approach: The code_file=True default was a classic "works for the common case" shortcut that became a security gap. Using file-extension heuristics to choose the right redaction mode is far more robust.
• Comprehensive extension set — 45+ source-code extensions covering all major languages. The set includes .sql, .pyi, .pyx, .vue, .svelte, .erb, and shell variants — this is more thorough than needed for the immediate bug and future-proof.
• frozenset + lowercase extension — efficient lookup, case-insensitive, immutable (won't accidentally grow at runtime)
• code_file=False by inference — all non-source-code files (.env, .yaml, .json, .toml, .ini, .cfg, .properties, .conf, .md, .txt, etc.) naturally get full redaction
• Two call sites updated — both read_file and search_files now use the same heuristic, maintaining consistency
• m.path in search_files — correctly uses the match result's file path (not the search path parameter)
• All 29 existing tests pass — no regressions

💡 Suggestions

1. Consider adding .tf (Terraform), .tfvars, .hcl, .dockerfile, and .Dockerfile to the extension set — these are configuration-as-code files that can contain secret-like constants (e.g., var.api_key = "test")
2. Add a test that explicitly verifies a .env file content is fully redacted while a .py file with MAX_TOKENS=3000 is not — this PR body states it works but adding a regression test would prevent future regressions
3. Consider edge case: a file named Dockerfile (no extension) — os.path.splitext("Dockerfile") returns ("Dockerfile", ""), so it would get code_file=False. This is fine since Dockerfiles can contain ENV assignments with secrets, but worth documenting

Testing Completeness

• 29/29 existing tests pass
• 155 redact-related tests pass, 3 skipped
• Manual verification with .env vs .py files

---

Reviewed by Hermes Agent

[tonydwb]

Code Review Summary: PR #35246

Verdict: APPROVE ✅

Overall: Precise, well-scoped fix with excellent PR description (before/after table, rationale, edge case documentation).

✅ Looks Good

• Correct approach: The code_file=True default was a classic "works for the common case" shortcut that became a security gap. Using file-extension heuristics to choose the right redaction mode is far more robust.
• Comprehensive extension set — 45+ source-code extensions covering all major languages. The set includes .sql, .pyi, .pyx, .vue, .svelte, .erb, and shell variants — this is more thorough than needed for the immediate bug and future-proof.
• frozenset + lowercase extension — efficient lookup, case-insensitive, immutable (won't accidentally grow at runtime)
• code_file=False by inference — all non-source-code files (.env, .yaml, .json, .toml, .ini, .cfg, .properties, .conf, .md, .txt, etc.) naturally get full redaction
• Two call sites updated — both read_file and search_files now use the same heuristic, maintaining consistency
• m.path in search_files — correctly uses the match result's file path (not the search path parameter)
• All 29 existing tests pass — no regressions

💡 Suggestions

1. Consider adding .tf (Terraform), .tfvars, .hcl, .dockerfile, and .Dockerfile to the extension set — these are configuration-as-code files that can contain secret-like constants (e.g., var.api_key = "test")
2. Add a test that explicitly verifies a .env file content is fully redacted while a .py file with MAX_TOKENS=3000 is not — this PR body states it works but adding a regression test would prevent future regressions
3. Consider edge case: a file named Dockerfile (no extension) — os.path.splitext("Dockerfile") returns ("Dockerfile", ""), so it would get code_file=False. This is fine since Dockerfiles can contain ENV assignments with secrets, but worth documenting

Testing Completeness

• 29/29 existing tests pass
• 155 redact-related tests pass, 3 skipped
• Manual verification with .env vs .py files

---

Reviewed by Hermes Agent

[tonydwb]

Code Review Summary

Verdict: Approved ✅

Overall

Smart, focused security fix. The code_file=True blanket was causing secrets in non-code files to bypass proper redaction, since env-assignment and JSON-field regex patterns were skipped globally. The fix introduces a targeted source-code extension check so config files get full coverage.

What's Good

• Correct diagnosis: The issue is clearly identified — redact_sensitive_text(code_file=True) was called for ALL files, bypassing ENV/JSON patterns everywhere
• Comprehensive extension set: Covers 30+ source-code extensions including shell scripts (.sh, .bash), SQL, and frontend frameworks
• Dual-path fix: Both read_file_tool and search_tool are updated (which use different call paths)
• Minimal diff: Only 53 lines added, no architecture changes
• Backward-compatible: Source files keep the same behavior; only non-code files change
• Well-documented: Clear docstring explaining why source code skips ENV patterns, and the before/after table in PR body is excellent

No Issues Found

• The _SOURCE_CODE_EXTENSIONS set covers language extensions comprehensively
• lower() normalization prevents case-sensitivity edge cases
• os.path.splitext correctly handles paths with multiple extensions (e.g., .spec.ts stays .ts)
• No existing tests break (all 155 redact tests pass)

---

Reviewed by Hermes Agent

[teknium1]

Thanks for identifying the non-code-file redaction gap. The premise remains present on current main: file_read=True forces code_file=True in agent/redact.py:541-542, so ENV/JSON assignment passes remain skipped for file-tool output.

Problems

• The proposed calls replace current main's file_read=True mode. That would discard the non-reusable prefix-secret sentinel introduced in de928bccd and restore reusable head/tail masks for config-file credentials (agent/redact.py:464-488, 541-547).
• Current main has three file-content paths—tools/file_tools.py:1279, 1413, and 2003—while this older patch only addresses the two original paths.
• No regression tests cover the path-sensitive behavior.

Suggested changes

• Preserve file_read=True semantics while designing a path-aware assignment-redaction mode, and cover all three current call sites.
• Add end-to-end file-tool tests for opaque non-source secrets, source-code false positives, and prefix-secret sentinels.

Automated hermes-sweeper review.

[teknium1]

Current main uses file_read=True for all file-content paths (tools/file_tools.py:1279,1413,2003), which enables the non-reusable prefix-secret sentinel in agent/redact.py:541-547. Replacing it with code_file=... here would restore reusable head/tail masks for config credentials; preserve the file_read security behavior when rebasing this change.