fix(ci): stabilize main test suite regressions by stephenschoettler · Pull Request #17660 · NousResearch/hermes-agent

stephenschoettler · 2026-04-29T23:20:45Z

What does this PR do?

Stabilizes the Hermes Agent main test suite so blocked PRs can be validated against a green base again.

The failures were mostly caused by host state, import-order state, and xdist ordering leaks. This PR hardens those paths so tests do not depend on a developer machine profile, local terminal state, current environment variables, TERM, or stale process-global state from earlier tests.

It also fixes a real streaming interrupt race surfaced by CI: if the worker thread observes _interrupt_requested before the outer polling loop does, the interrupt is now propagated through the normal result channel instead of becoming an unhandled thread exception.

This intentionally keeps the Codex Responses tool-call history fix out of this branch. That work remains in PR #17645.

Related Issue

No linked GitHub issue.

Replacement path for the closed, unmerged #17618 CI-stabilization PR.

Related PR: #17645 remains separate for Codex Responses tool-call history sanitization.

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
🔒 Security fix
📝 Documentation update
✅ Tests (adding or improving test coverage)
♻️ Refactor (no behavior change)
🎯 New skill (bundled or hub)

Changes Made

run_agent.py
- Propagate fast streaming interrupts raised inside the worker thread before the outer polling loop observes them.
tests/conftest.py
- Reset process-global logger state and terminal/environment caches between tests.
tools/environments/local.py
- Cache POSIX process group ids so interrupt cleanup can still kill orphaned subprocess groups after a shell wrapper exits.
tests/tools/test_local_interrupt_cleanup.py
- Harden process-group cleanup assertions and add diagnostics for surviving subprocesses.
tests/tools/test_tirith_security.py
- Make HOME fallback assertions compute expected paths under the same cleared environment as the code under test.
tests/hermes_cli/test_config_env_expansion.py, tests/hermes_cli/test_container_aware_cli.py, tests/hermes_cli/test_web_server.py, tests/test_cli_skin_integration.py
- Patch imported function globals directly where module reloads can make string-target monkeypatches hit stale module objects.
tests/hermes_cli/test_pty_bridge.py, tests/hermes_cli/test_web_server.py
- Avoid tput and TERM assumptions in PTY resize coverage by querying the TTY ioctl directly.
tests/agent/test_minimax_provider.py
- Update MiniMax normalization expectations so non-Anthropic model names keep canonical dot separators.
Gateway, CLI, TUI gateway, MCP, TTS/STT, clipboard, approval heartbeat, and file-tool test updates
- Remove CI-only assumptions around host state, provider env, redaction/quiet mode, process detection, websocket resize behavior, and temp HOME/HERMES_HOME isolation.

How to Test

Run the streaming interrupt regression suite:

python -m pytest tests/run_agent/test_stream_interrupt_retry.py -q --tb=short -o addopts=''

Result:

3 passed

Run the agent/context focused suite:

python -m pytest tests/run_agent/test_stream_interrupt_retry.py tests/run_agent/test_run_agent.py tests/run_agent/test_provider_parity.py tests/agent/test_context_compressor.py -q --tb=short -o addopts=''

Result:

468 passed

Run targeted CLI/tool regressions:

python -m pytest tests/hermes_cli/test_gateway_service.py::TestLaunchdServiceRecovery::test_launchd_start_reloads_on_kickstart_exit_code_113 tests/hermes_cli/test_claw.py::TestCmdCleanup::test_dry_run_lists_dirs -q --tb=short -o addopts=''
python -m pytest tests/tools/test_clipboard.py::TestIsWsl -q --tb=short -n auto -o addopts=''
python -m pytest -q -o addopts='' --tb=short tests/agent/test_minimax_provider.py::TestMinimaxPreserveDots tests/tools/test_local_interrupt_cleanup.py tests/tools/test_tirith_security.py::TestHermesHomeIsolation
python -m pytest -q -o addopts='' --tb=short -n auto tests/hermes_cli/test_config_env_expansion.py tests/hermes_cli/test_container_aware_cli.py tests/hermes_cli
python -m pytest -q --tb=short -o addopts='' tests/hermes_cli/test_pty_bridge.py tests/hermes_cli/test_web_server.py::TestReloadEnv tests/hermes_cli/test_web_server.py::TestPtyWebSocket tests/test_cli_skin_integration.py

Results:

2 passed
5 passed
19 passed
3207 passed, 7 skipped
40 passed

Run the CI-like full suite in an isolated environment:

TMP=$(mktemp -d)
mkdir -p "$TMP/home" "$TMP/hermes"
HOME="$TMP/home" HERMES_HOME="$TMP/hermes" CI=true \
  OPENROUTER_API_KEY='' OPENAI_API_KEY='' NOUS_API_KEY='' \
  python -m pytest tests/ -q --ignore=tests/integration --ignore=tests/e2e --tb=short -n auto -o addopts=''

Result:

17831 passed, 40 skipped

Run syntax and whitespace checks:

git diff --check
python -m compileall -q <touched files>

Validation Status

Rebased onto current origin/main (51b44b6e3) to clear merge conflicts and avoid stale-base supply-chain scan noise.
Conflict resolution preserved current main behavior where it had newer equivalents and kept the CI hardening changes where still needed.

Syntax and whitespace checks: passed.

/home/w0lf/.hermes/hermes-agent/venv/bin/python -m compileall -q     agent/copilot_acp_client.py agent/redact.py gateway/platforms/base.py     gateway/run.py gateway/session.py hermes_cli/doctor.py hermes_cli/main.py     hermes_cli/web_server.py run_agent.py tests/agent/test_minimax_provider.py     tests/hermes_cli/test_auth_commands.py tests/hermes_cli/test_setup.py     tests/test_tui_gateway_server.py tests/tools/test_clipboard.py     tools/environments/local.py tools/file_tools.py tools/mcp_tool.py     tools/transcription_tools.py tools/tts_tool.py tui_gateway/server.py
git diff --check origin/main...HEAD

Focused regression suite after latest rebase: 359 passed.

/home/w0lf/.hermes/hermes-agent/venv/bin/python -m pytest     tests/agent/test_minimax_provider.py     tests/hermes_cli/test_auth_commands.py     tests/hermes_cli/test_setup.py     tests/test_tui_gateway_server.py     tests/tools/test_clipboard.py     tests/run_agent/test_stream_interrupt_retry.py     -q --tb=short -o addopts=''

CI-like full local suite with isolated HOME/HERMES paths and TERM unset after latest rebase: 17870 passed, 43 skipped.

/home/w0lf/.hermes/hermes-agent/venv/bin/python -m pytest     tests/ -q --ignore=tests/integration --ignore=tests/e2e --tb=short -n auto

GitHub checks on the latest rebased head: all passing (check-attribution, nix (ubuntu-latest), nix-lockfile-check, Scan PR for critical supply chain risks, test, nix (macos-latest), e2e).
Manual interactive testing: not run separately. This is CI stabilization work; the affected terminal, process, path, banner, and streaming interrupt paths are covered by the automated tests above.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
I searched for existing PRs to make sure this isn't a duplicate
My PR contains only changes related to this fix/feature (no unrelated commits)
I've run the CI test suite and all tests pass
I've added tests for my changes (required for bug fixes, strongly encouraged for features)
I've tested on my platform: Linux, Python 3.11

Documentation and Housekeeping

I've updated relevant documentation (README, docs/, docstrings) or N/A
I've updated cli-config.yaml.example if I added/changed config keys or N/A
I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows or N/A
I've considered cross-platform impact (Windows, macOS) per the compatibility guide or N/A
I've updated tool descriptions/schemas if I changed tool behavior or N/A

For New Skills

N/A. This PR does not add a skill.

Screenshots / Logs

Previous CI failure fixed by the final runtime patch:

FAILED tests/run_agent/test_stream_interrupt_retry.py::TestStreamInterruptBeforeRetry::test_interrupt_before_first_attempt
Failed: DID NOT RAISE <class 'InterruptedError'>
[gateway-crash] thread ... raised InterruptedError: Agent interrupted before stream retry

stephenschoettler · 2026-04-30T04:32:32Z

@alt-glitch could you take a look at the priority on this PR?

It is currently labeled P3, but I think the operational impact is closer to P2 or unblocker.

The code scope is test-suite stabilization, but the reason it matters is that it restores a trustworthy green base for the rest of the open PR queue. Without this, other PRs are harder to review because failures can come from the broken baseline instead of the PR under review.

It is also time-sensitive because main is moving quickly. Main already advanced while this PR's CI was running once, which made the PR stale and caused a temporary stale-base supply-chain scan failure. I rebased, reran validation, and the PR is currently clean, mergeable, and green.

Could we upgrade the priority or mark it as an unblocker so it gets reviewed before the follow-up queue?

…#17801) PR #17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit b767b5e) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 435b2c1) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 20d0bfd) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

* fix: stabilize main test suite regressions * test(agent): update MiniMax normalization expectation * test: stabilize remaining CI assertions * test: harden config helper monkeypatching * test: harden CI-only assertions * fix(agent): propagate fast streaming interrupts

…search#17660 (NousResearch#17801) PR NousResearch#17660 landed a sweep of CI fixes but left three loose ends: 1. tests/cli/test_cli_loading_indicator.py::test_reload_mcp_sets_busy_state_ and_prints_status — /reload-mcp gained a prompt-cache-invalidation confirmation (commit 4d7fc0f) that was never wired into this test. The test exercises the loading-indicator path, so pre-approve via config and go straight into _reload_mcp(). 2. tools/mcp_tool.py _make_tool_handler — the added getattr(server, '_rpc_lock', None) + 'skip the lock if missing' branch is inconsistent with four sibling call sites that still direct-access server._rpc_lock. The lock is guaranteed by MCPServerTask.__init__; falling through to an unlocked session.call_tool would silently serialize-strip RPCs if the guard ever triggered. Restore direct access. 3. tui_gateway/server.py _messages_as_conversation — the helper existed only to catch 'TypeError: include_ancestors unexpected' from mocked SessionDBs that don't actually exist. The real SessionDB.get_messages_as_conversation has accepted include_ancestors since introduction, and every test FakeDB in the repo already declares the kwarg. Remove the shim, inline the two call sites.

stephenschoettler changed the title ~~fix: stabilize main test suite regressions~~ fix(ci): stabilize main test suite regressions Apr 30, 2026

stephenschoettler force-pushed the fix/main-test-suite-regressions branch from 5b4a5fe to e1df9c8 Compare April 30, 2026 03:44

stephenschoettler added 6 commits April 29, 2026 20:52

fix: stabilize main test suite regressions

3bbd512

test(agent): update MiniMax normalization expectation

5cbcf8c

test: stabilize remaining CI assertions

a4726fa

test: harden config helper monkeypatching

da16298

test: harden CI-only assertions

f416659

fix(agent): propagate fast streaming interrupts

279b83c

stephenschoettler force-pushed the fix/main-test-suite-regressions branch from e1df9c8 to 279b83c Compare April 30, 2026 03:55

teknium1 merged commit f73364b into NousResearch:main Apr 30, 2026
7 checks passed

teknium1 mentioned this pull request Apr 30, 2026

fix: clean up defensive shims and finish CI stabilization from #17660 #17801

Merged

This was referenced May 1, 2026

fix(agent): sanitize Codex tool-call history summaries #18176

Merged

fix(agent): sanitize Codex tool-call history summaries #17645

Closed

konsisumer mentioned this pull request May 2, 2026

fix: prevent vision_analyze timeouts on larger local images #8168

Closed

Navigation Menu

Uh oh!

fix(ci): stabilize main test suite regressions - #17660

fix(ci): stabilize main test suite regressions#17660
teknium1 merged 6 commits into
NousResearch:mainfrom
stephenschoettler:fix/main-test-suite-regressions

stephenschoettler commented Apr 29, 2026 •

edited

Loading

Uh oh!

stephenschoettler commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

stephenschoettler commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Validation Status

Checklist

Code

Documentation and Housekeeping

For New Skills

Screenshots / Logs

Uh oh!

stephenschoettler commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stephenschoettler commented Apr 29, 2026 •

edited

Loading