test: use subprocesses for each test file

[ethernet8023]

What does this PR do?

Replaces fragile manual _reset_module_state autouse fixtures + pytest-xdist with per-file process isolation — each test file runs in a freshly-spawned python -m pytest <file> subprocess via scripts/run_tests_parallel.py.

Cross-file module-level state leakage (module-level dicts, ContextVars, caches) is impossible: every file gets a clean interpreter. Intra-file ordering is the test author's responsibility.

Why per-file, not per-test?

Per-test subprocess isolation is the obvious-sounding answer but doesn't survive the math:

• 17k tests × ~250ms spawn cost = 70min CPU minimum, even on infinite cores (Amdahl wall)
• xdist + a per-test isolation plugin compound the coordination overhead
• previous attempts ran 30+ minutes and never finished in CI

Per-file is the actual isolation boundary that matters:

• cross-file module-level state leakage is what _reset_module_state existed to clean up — fixed for free by giving each file a fresh interpreter
• intra-file ordering is the test author's responsibility; if test A in foo.py mutates state test B in foo.py reads, that's a real bug that bites anyone running pytest tests/foo.py directly
• ~1125 files × ~250ms = ~3.5min of pure spawn overhead — fits the budget with room to spare

Why drop xdist?

xdist's persistent worker pool accumulates state across files within a worker, which is exactly the leakage class we want to prevent. xdist also brings --dist=load vs loadfile vs loadscope vs worksteal, --max-worker-restart semantics, and an internal control plane — all of which we don't need when the unit of work is "run pytest on one file in a fresh process." A ThreadPoolExecutor of N workers each calling subprocess.run([python, -m, pytest, file]) is ~60 lines, has cleaner semantics, and gives stronger guarantees.

Performance

Local (16-core box, full suite ~1125 files / ~17k tests):

approach	wall time
old: xdist -n auto + per-test spawn isolation plugin	13+ min, never finished
new: per-file subprocess.run with os.cpu_count() workers	~5 min

CI (4-core ubuntu-latest): ~8 min, well within the 60-min job budget.

Type of Change

• 🐛 Bug fix (eliminates the cross-file state-leakage flake class)
• ✅ Tests (test infrastructure rework + ~30 test fixes across 14 files)
• ♻️ Refactor (no behavior change for tests themselves; only how they are scheduled)

Surfaced and Fixed Bugs

Switching to per-file isolation exposed ~30 previously-hidden test failures across 14 files. Under xdist, these tests passed only because other files' import side effects populated shared module-level state. All are now fixed:

Tool registry not populated in isolation (9 files, ~16 tests)

These tests accessed the tool or web-search-provider registry before discover_builtin_tools() had run. Under xdist, a co-worker's earlier import populated the registry as a side effect.

File	Fix
tests/tools/test_video_generation_tool_surface_matrix.py	discover_builtin_tools() call before registry access
tests/tools/test_web_providers_brave_free.py	Shared autouse fixture (tests/tools/conftest.py) registering all 8 bundled web providers + is_safe_url/check_website_access mocks
tests/tools/test_web_providers_ddgs.py	Same shared fixture + SSRF mocks
tests/tools/test_web_providers_searxng.py	Same shared fixture + SSRF mocks
tests/tools/test_web_providers.py	Same shared fixture
tests/tools/test_website_policy.py	Same shared fixture
tests/tools/test_web_tools_tavily.py	Same shared fixture across 3 dispatch test classes
tests/tools/test_discord_tool.py	Cache invalidation in setup/teardown
tests/tools/test_homeassistant_tool.py	invalidate_check_fn_cache() before registry queries

Stale check_fn / tool-defs cache (1 file, 2 tests)

test_kanban_guidance_not_in_normal_prompt cached check_fn(kanban_show) → False. Next test set HERMES_KANBAN_TASK but never invalidated the cache.

File	Fix
tests/tools/test_kanban_tools.py	invalidate_check_fn_cache() + _clear_tool_defs_cache() in both kanban guidance tests

Module-level state pollution (4 files, ~12 tests)

File	Root Cause	Fix
tests/agent/test_auxiliary_client.py	_aux_unhealthy_until / _aux_unhealthy_logged_at dicts persisted across test boundaries	Autouse fixture clearing both dicts + unhealthy cache before each test
tests/agent/test_skill_commands.py	patch.dict(os.environ, {"HERMES_SESSION_PLATFORM": "telegram"}) — ContextVar takes precedence over os.environ	set_session_vars(platform="telegram") + clear_session_vars() in finally block
tests/gateway/test_dm_topics.py	sys.modules.setdefault for telegram mock shadowing attributes; cached gateway.platforms.telegram import	Overwrite sys.modules[name] (not setdefault), register telegram.constants as separate module, del sys.modules["gateway.platforms.telegram"]
tests/tools/test_terminal_tool_requirements.py	Duplicate class TestTerminalRequirements: (IndentationError) + stale tool-defs cache	Remove duplicate class declaration, add autouse _clear_caches fixture

Timing flakes (1 file, 1 test)

File	Root Cause	Fix
tests/plugins/test_achievements_plugin.py	Background scan thread completes before evaluate_all() returns stale data (10 fake sessions too fast)	scan_delay=2.0 on _FakeSessionDB so background thread can't win the race

Other fixes

File	Fix
tests/tools/test_send_message_tool.py	Skip when telegram package not installed
tests/tools/test_browser_supervisor.py	Handle missing worker_id (no xdist)
tests/gateway/conftest.py	Major rewrite: gateway mock setup no longer depends on xdist worker fixtures
tests/tools/test_approval_plugin_hooks.py	Isolation fixes for tool registry
tests/tools/test_write_deny.py	Isolation fixes for tool registry
tests/hermes_cli/test_pty_bridge.py	Isolation compat
tests/plugins/web/test_web_search_provider_plugins.py	Provider registration + xai mock

Other Changes

hermes_cli/main.py — py_compile race fix

_validate_critical_files_syntax used to write .pyc into the source tree's __pycache__/, which races with concurrent test workers. Now uses tempfile.TemporaryDirectory so the compiled output is isolated and discarded.

hermes_cli/profiles.py — NixOS rmtree fix

shutil.rmtree(profile_dir) fails on NixOS where profile dirs contain read-only copies from the immutable Nix store. Added a _make_writable onexc/onerror handler that adds +w on PermissionError (both the path itself and its parent). Compatible with Python 3.11 (onerror) and 3.12+ (onexc).

CI — ripgrep via prebuilt tarball

Replaced sudo apt-get install -y ripgrep (~4 min) with a pinned, sha256-verified prebuilt binary download (~5s). Eliminates the apt-get update cold-start penalty.

.gitignore — .pytest-cache/

Added .pytest-cache/ (emitted by the parallel runner's per-file subprocesses).

Files Changed (summary)

Added:

• scripts/run_tests_parallel.py (~650 lines) — per-file subprocess runner with ThreadPoolExecutor, captured stdout, live progress, exit-code-5-as-pass handling, per-file test-level pass/fail counts
• tests/test_run_tests_parallel.py (187 lines) — self-validating tests for the runner
• tests/tools/conftest.py (50 lines) — shared autouse fixture for web provider registry + SSRF mocks

Removed:

• pytest-xdist==3.8.0 from dev deps
• -n auto from pyproject.toml addopts
• ~150 lines of _reset_module_state + related autouse fixtures from tests/conftest.py

Net: +1695 / −583 across 35 files

How to Test

# Full suite, default parallelism = cpu_count*2
scripts/run_tests.sh

# Cap workers
scripts/run_tests.sh -j 4

# Single file, direct pytest (no parallel runner overhead)
scripts/run_tests.sh tests/agent/test_async_utils.py

# Pytest passthrough
scripts/run_tests.sh -- --tb=long -v -k 'TestFoo'