fix: tolerate non-utf8 filenames in file discovery

[dso2ng]

Summary

• make CLI project file discovery tolerate non-UTF-8 filenames during bare @ completion
• apply the same filesystem-safe decoding to context reference folder discovery
• add regression tests covering non-UTF-8 filenames in both code paths

Why

Some repositories contain filenames that are valid on the filesystem but not valid UTF-8. Hermes should not crash when scanning those trees for @ file completion or folder context references.

Root cause

subprocess.run(..., text=True) decoded rg --files output as strict UTF-8. If the working tree contained non-UTF-8 filenames, Hermes raised UnicodeDecodeError while building completions or folder listings.

Behavior before

• bare @ completion could crash on mixed-encoding trees
• folder context discovery could fail when rg --files returned non-UTF-8 names

Behavior after

• file discovery subprocess calls use text=False
• each output line is decoded with os.fsdecode(...), preserving filesystem semantics via surrogateescape
• Hermes keeps normal behavior for UTF-8 paths while tolerating mixed-encoding trees without crashing

How to test

• python -m pytest tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py -o 'addopts=' -q
• In a repository containing at least one non-UTF-8 filename, verify that:
  • bare @ completion no longer crashes
  • folder/context reference expansion still works

Platforms tested

• Linux (local Hermes development environment on omarchy)

Related issues / PRs

• Related to OpenClaw migration crashes with UnicodeDecodeError on Windows (non-UTF-8 locale) #8901 and [Bug]: Non-UTF-8 MEMORY.md raises uncaught UnicodeDecodeError — agent starts with empty memory #10879 as another example of strict UTF-8 assumptions causing UnicodeDecodeError in real-world environments.
• Similar in problem family to fix: handle non-UTF-8 files in OpenClaw migration script #8898 / fix(migration): handle non-UTF-8 files in OpenClaw migration #9145, which harden OpenClaw migration against non-UTF-8 files; this PR is specifically scoped to CLI file discovery and context reference expansion.
• Distinct from Windows/stdin/config encoding fixes like fix(cli): force UTF-8 stdio bootstrap #10979, fix: add UTF-8 encoding for Windows compatibility #11990, and fix(cli): load Hermes dotenv in mini_swe and sample_and_compress #8415 because this bug is about filesystem filename bytes returned from subprocess file listing, not about stdio or config file decoding.

Notes

• This PR is intentionally scoped to filesystem-safe decoding in file discovery and the corresponding regression coverage.

[dso2ng]

Maintainer context: I did a repo search for related utf-8 / UnicodeDecodeError work before narrowing this PR.

Related but distinct items:

• OpenClaw migration crashes with UnicodeDecodeError on Windows (non-UTF-8 locale) #8901, fix: handle non-UTF-8 files in OpenClaw migration script #8898, and fix(migration): handle non-UTF-8 files in OpenClaw migration #9145 are migration-path fixes for non-UTF-8 files during OpenClaw migration.
• [Bug]: Non-UTF-8 MEMORY.md raises uncaught UnicodeDecodeError — agent starts with empty memory #10879 is another strict UTF-8 assumption bug, but in memory loading.
• fix(cli): force UTF-8 stdio bootstrap #10979 / fix: add UTF-8 encoding for Windows compatibility #11990 / fix(cli): load Hermes dotenv in mini_swe and sample_and_compress #8415 are more about stdio / config / Windows encoding bootstrap and file decoding.

This PR is intentionally scoped to a different failure surface:

• filesystem filename bytes returned from subprocess-based file listing (rg --files),
• bare @ completion in the CLI,
• and folder/context reference expansion.

So it is in the same bug family, but it is not trying to reopen the migration or Windows-stdio fixes.

[dso2ng]

Maintainer note: this fork-based PR has focused local verification already completed:

• python -m pytest tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py -o 'addopts=' -q

Current blocker appears to be GitHub Actions approval for fork workflows: the PR checks are showing action_required with zero-job runs, so CI has not actually started yet.

If a maintainer approves / runs the PR workflows in the GitHub UI, I can follow up on any real CI failures after that.

[dso2ng]

@kshitijk4poor @teknium1 friendly ping on this bugfix as well: it is a small, focused fix for non-UTF-8 filename handling in CLI file discovery/context expansion, with regression coverage already in place.

Current blocker seems to be fork-workflow approval (action_required / 0s runs). If you approve/run the PR workflows in the GitHub UI, I can follow up on any actual CI failures once the checks really start.

[dso2ng]

Refresh update: I merged current origin/main into this branch and resolved the hermes_cli/commands.py conflict while preserving the byte-safe filesystem decoding path.

Updated head: ed5cdfd62

Focused local verification on the refreshed branch:

• /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m pytest tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py -o 'addopts=' -q -> 42 passed
• /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m py_compile hermes_cli/commands.py agent/context_references.py tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py
• git diff --check

The PR is mergeable again locally / via GitHub metadata. The current GitHub Actions runs on the refreshed head still show action_required with zero jobs, so they appear to be waiting for maintainer approval to run fork workflows.

[dso2ng]

Refresh update: I merged current origin/main into this branch and resolved the tests/agent/test_context_references.py conflict while preserving both the non-UTF-8 filename regression coverage and the newer upstream binary-file context-reference test.

Updated head: 120d83689

Focused local verification on the refreshed head:

• git diff --check -> passed
• python -m py_compile agent/context_references.py hermes_cli/commands.py tests/agent/test_context_references.py tests/hermes_cli/test_path_completion.py -> passed
• python -m pytest tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py -o 'addopts=' -q -> 43 passed, 1 warning
• public diff non-ASCII guard -> non_ascii_added_lines=0

GitHub now reports the PR as mergeable again. The new workflow runs are still at the fork workflow approval gate (action_required with zero jobs), not a code/test failure.

[teknium1]

Thanks for the focused filesystem-decoding fix. The context-reference premise is present on current main: agent/context_references.py:533-547 runs rg --files with text=True, and the proposed byte/os.fsdecode conversion is the right direction.

Problems

• tests/hermes_cli/test_path_completion.py:187 does not prove the classic CLI fix. Current main uses errors="replace" at hermes_cli/commands.py:1662-1667, so an unusable @file:bad-�.txt completion still passes the startswith("@file:bad-") assertion. Assert that the returned filename suffix round-trips through os.fsencode to b"bad-\xff.txt".
• The matching TUI path remains affected: ui-tui/src/hooks/useCompletion.ts:34-37 routes @ completion to complete.path, while tui_gateway/server.py:12361,12380 UTF-8 replacement-decodes git path output. Please include byte-safe decoding and regression coverage there as well.

Automated hermes-sweeper review.

[teknium1]

This only checks the prefix. Current main decodes rg output with errors="replace" (hermes_cli/commands.py:1662-1667), so it produces @file:bad-�.txt and this assertion still passes. Assert an os.fsencode() round-trip of the path suffix to prove surrogateescape preservation.

[dso2ng]

Thanks — addressed on the current head dc44cef04.

• The CLI completion regression now asserts the returned filename suffix round-trips with os.fsencode(...) == b"bad-\\xff.txt", so UTF-8 replacement decoding cannot satisfy the test.
• The matching TUI path now uses byte-safe filesystem decoding for Git -z output, with regression coverage in tests/test_tui_gateway_server.py.

Focused verification on the updated head:

python -m pytest tests/hermes_cli/test_path_completion.py tests/agent/test_context_references.py tests/test_tui_gateway_server.py -o 'addopts=' -q
368 passed

Could you please re-review the updated head?