fix(gateway): bound hygiene compression failures

[westkite1201]

Summary

• add a bounded timeout for gateway pre-agent hygiene compression
• add a per-session cooldown after hygiene compression timeout/abort so repeated messages do not block on the same broken auxiliary backend
• document the new compression.hygiene_timeout_seconds and compression.hygiene_failure_cooldown_seconds knobs

Why

Long-running gateway sessions can become oversized and trigger hygiene compression before handling an inbound message. If the auxiliary compression backend is depleted, misconfigured, or slow, the gateway can appear stuck by repeatedly blocking on that pre-agent recovery path.

Tests

• python -m pytest tests/gateway/test_session_hygiene.py -q
• python -m pytest tests/gateway/test_session_hygiene.py tests/gateway/test_compress_command.py::test_compress_command_passes_session_db_and_persists_rotated_session tests/agent/test_compression_rotation_state.py tests/agent/test_compression_concurrent_fork.py -q
• python -m py_compile gateway/run.py hermes_cli/config.py
• git diff --check

[alt-glitch]

This was generated by AI during triage.

Related: #49905 bounds the agent context-compression summary stall; this PR bounds the gateway pre-agent hygiene compression path. Same compression-stall family, different code path — flagging the cluster so a maintainer can decide whether to consolidate.

[westkite1201]

Thanks for flagging. Agreed: this is the same compression-stall family as #49905, but it intentionally covers the gateway pre-agent hygiene path rather than the agent's in-loop context-compressor summary path.

I'm happy to rebase/consolidate if maintainers prefer that both timeout/fallback paths land together, but keeping this PR scoped to gateway hygiene avoids changing the agent compression wrapper while still preventing inbound gateway messages from repeatedly blocking on a slow/depleted auxiliary compression backend.

[tonydwb]

Code Review Summary

Verdict: Comment (core gateway change)

This PR adds 293 lines (22KB diff) to gateway/run.py, introducing timeout and cooldown settings for hygiene compression failures. The changes are well-documented and follow the existing config parsing pattern.

However, this modifies core gateway behavior (compression failure handling) and the 22KB diff size includes significant context. Key considerations:

• Timeout and cooldown settings affect all conversations
• Default values (30s timeout, 300s cooldown) need validation against production behavior
• Config parsing follows existing patterns but touches critical path

The implementation is clean, but the gateway is a critical component. Recommend human review of the timeout/cooldown defaults and their interaction with existing compression logic.

---

Reviewed by Hermes Agent

[Kinkoolino-Hermes]

Small consolidation note after related compaction fixes landed elsewhere: this PR still appears to cover a distinct path.

The merged Codex/max-output and in-loop compaction fixes address agent-side recovery/flush behavior. This PR is about the gateway pre-agent hygiene compression path, where a slow or broken auxiliary compression backend can block before the agent turn is even reached.

So the useful review question is whether Hermes wants a bounded timeout/cooldown for that gateway hygiene path, independent of the agent-context compression recovery fixes.

[westkite1201]

Rebased this branch onto current main and resolved the gateway hygiene conflict. The conflict resolution keeps the PR's timeout behavior, but uses the current main off-loop cleanup path when the hygiene compression worker did not time out.

Validation run locally:

• uv run --with pytest-timeout pytest tests/gateway/test_session_hygiene.py -q -k 'hygiene_timeout or failure_cooldown'\n- uv run --with pytest-timeout pytest tests/gateway/test_session_hygiene.py -q\n- uv run ruff check gateway/run.py hermes_cli/config.py tests/gateway/test_session_hygiene.py\n- python3 -m py_compile gateway/run.py hermes_cli/config.py tests/gateway/test_session_hygiene.py\n- git diff --check\n- python3 scripts/check-windows-footguns.py --diff origin/main\n\nGitHub now reports the PR as mergeable again.

[teknium1]

Thanks for targeting the distinct gateway pre-agent hygiene stall path; current main still awaits this compression before _run_agent at gateway/run.py:11316-11322.

Problems

• The timeout path starts executor work at gateway/run.py:10991 and then intentionally skips the only cleanup at gateway/run.py:11173-11176. The current cleanup routine shuts down the memory provider and closes tool/client resources (gateway/run.py:6037-6076), so a timed-out worker that eventually returns leaks its temporary agent.
• Current main now binds hygiene compaction to the live SessionDB (gateway/run.py:11278-11309) and starts _run_agent after hygiene failure handling (gateway/run.py:11632-11650). Salvage must not let a timed-out worker continue mutating that session concurrently with the new turn.

Suggested changes

• Use a lifecycle boundary that stops/fences timed-out compression before the live turn proceeds, and ensure cleanup happens after worker completion.
• Add a SessionDB-bound timeout regression covering both no post-timeout compaction and eventual resource cleanup.

Automated hermes-sweeper review.

[teknium1]

wait_for() bounds the await, not this executor thread. The timeout path later skips cleanup entirely, so a worker that eventually returns leaks its temporary AIAgent; once salvaged onto current main, that worker can also race the SessionDB-bound hygiene compactor. Please fence/terminate completion before proceeding to the live turn and arrange cleanup after it is safe.

[teknium1]

Merged via #69866 (commit 2e9765b). Salvaged with worker fencing added: bounded await + cancellation-token no-op for late workers + temp-agent cleanup + lock-reacquire verification. Note: the commit was authored by @kshitijk4poor and lands with his authorship; PR-submission credit to you in the merged body.