Skip to content

fix(gateway): persist session hygiene compression-failure cooldown to state DB (#74136) - #74417

Closed
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/74136-compression-cooldown-persistent
Closed

fix(gateway): persist session hygiene compression-failure cooldown to state DB (#74136)#74417
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/74136-compression-cooldown-persistent

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Problem

The session hygiene compression path tracked its per-session failure cooldown in an in-memory dict (_hygiene_compression_failure_cooldowns) on the gateway process. When the gateway restarts, that dict is lost, so the same failing compression is immediately re-triggered on the next message - wedging session storage.

The user-visible symptom: after a compression failure (timeout, API error, abort), every message across all sessions gets the "session storage could not be written" error. Restarting the gateway does not recover - it clears the in-memory cooldown and the cycle repeats.

Root Cause

The state DB already has a persistent column sessions.compression_failure_cooldown_until and full read/write/clear methods (record_compression_failure_cooldown / get_compression_failure_cooldown / clear_compression_failure_cooldown in hermes_state.py). The in-conversation compression path (agent/context_compressor.py) already uses these persistent methods correctly. But the session hygiene path in gateway/run.py used its own in-memory dict instead - the two paths disagreed on durability.

Fix

  1. Cooldown check (line ~13836): Replace the in-memory dict lookup with await session_db.get_compression_failure_cooldown(session_id), consistent with the in-conversation path.
  2. Timeout cooldown write (line ~14020): Replace in-memory dict write with await session_db.record_compression_failure_cooldown(...).
  3. Abort cooldown write (line ~14186): Same replacement, with proper error text redaction before persistence.
  4. Tests (6 assertions across 4 test functions): Updated to assert against session_db._db mocks instead of the removed in-memory dict.

Fixes #74136

… state DB (NousResearch#74136)

The session hygiene compression path tracked its per-session failure
cooldown in an in-memory dict (self._hygiene_compression_failure_cooldowns).
When the gateway restarts, that dict is lost, so the same failing
compression is immediately re-triggered — wedging session storage.

The state DB already has a persistent column
(sessions.compression_failure_cooldown_until) and dedicated methods
(record_/get_/clear_compression_failure_cooldown) in hermes_state.py,
used by the in-conversation compression path (context_compressor.py)
but not by the session hygiene path.

Fix:
- Replace the in-memory dict check with
  session_db.get_compression_failure_cooldown() so the cooldown survives
  gateway restarts.
- Replace in-memory writes with
  session_db.record_compression_failure_cooldown() in both the timeout
  and abort paths.
- Clean up the abort-path error handling to eliminate redundant reads.
- Update tests to assert against session_db mock instead of the removed
  in-memory dict.

Fixes NousResearch#74136
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery area/compression Context compression and continuation sessions area/sessions Session lifecycle, resume, persistence, history sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages duplicate This issue or pull request already exists labels Jul 29, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #74251 — both persist the session-hygiene compression-failure cooldown through SessionDB so a gateway restart cannot immediately re-trigger the failing path.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for targeting the persistent cooldown mismatch. The premise remains valid on current main: hygiene still uses the process-local map at gateway/run.py:16041-16054, with writes at gateway/run.py:16225-16227 and gateway/run.py:16390-16393.

Problems

  • The updated tests do not provide the async facade the production code awaits. Production creates AsyncSessionDB(SessionDB()) at gateway/run.py:5782-5785; its callable forwarding is awaitable at hermes_state.py:8555-8569. The changed hygiene fixtures retain SimpleNamespace(_db=MagicMock()) (for example tests/gateway/test_session_hygiene.py:505), so the new outer-facade lookup/write raises AttributeError and is swallowed. The raw _db mock recorder is therefore not called.
  • The central restart guarantee lacks a SessionDB-backed regression.

Suggested changes

  • Wrap the test mock in AsyncSessionDB, or expose explicit outer AsyncMock lookup/record methods; configure inactive lookup values where compression should proceed.
  • Add a regression that persists a cooldown, constructs a fresh gateway DB facade over the same state DB, and verifies hygiene skips compression.

Automated hermes-sweeper review.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 30, 2026
@teknium1

teknium1 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Closing as a duplicate of #74251, which was submitted earlier with the same fix (swap the in-memory _hygiene_compression_failure_cooldowns dict for the DB-backed cooldown methods) and which we're salvaging for #74136 — adding a real AsyncSessionDB-backed restart regression test on top, since the existing mocks assert nothing against the awaited production path.

Your test-update intent will be folded into that salvage. Thanks — premise fully confirmed on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions area/sessions Session lifecycle, resume, persistence, history comp/gateway Gateway runner, session dispatch, delivery duplicate This issue or pull request already exists P1 High — major feature broken, no workaround sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Session hygiene compression cooldown is in-memory only — restarts re-trigger the same failing compression, wedging session storage

3 participants