fix(agent): persist compression backoff across resume + bound lease refresher (#54465) - #55499
Merged
kshitijk4poor merged 7 commits intoJun 30, 2026
Conversation
…DB blips Follow-up hardening on the salvaged NousResearch#54465 backoff persistence work. The lease refresher's loop treated ANY falsy refresh as a permanent stop (`if not refreshed: break`), conflating two distinct cases: - genuine lost-ownership (rowcount 0) — correct to stop, and - a one-off transient DB error (write contention that escapes _execute_write's retry budget) — which returned False identically. A single transient blip therefore killed the lease for the rest of a multi-minute compression call, silently reintroducing the exact 300s-TTL < ~361s-call expiry wedge the PR set out to fix. Changes: - _CompressionLockLeaseRefresher._run now tolerates a bounded run of consecutive failures (_MAX_CONSECUTIVE_REFRESH_FAILURES = 3) before giving up the lease; a recovered tick resets the counter. Worst-case extra hold is cap * refresh_interval, still bounded by the acquirer's TTL. - Replace the two remaining silent `except Exception: pass` arms in the compression-failure-cooldown persist/clear helpers with debug logging, for parity with their sqlite3.Error sibling arms (a non-sqlite bug was invisible). - Document the join(timeout=1.0) quiesce bound in stop(). - Add 3 regression tests: single-blip tolerance, persistent-failure stop at the cap, and refresh-raising tolerance.
alt-glitch
added
type/bug
teknium1
added a commit
that referenced
this pull request
Jun 30, 2026
…54465) (#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of #54465 (the resume-wedge persistence half landed in #55499).
dtera
pushed a commit
to dtera/hermes-agent
that referenced
this pull request
Jul 1, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
caozuohua
pushed a commit
to caozuohua/hermes-agent
that referenced
this pull request
Jul 2, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
Jasper6439
pushed a commit
to Jasper6439/hermes-agent
that referenced
this pull request
Jul 5, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
habarmc1223-sudo
pushed a commit
to habarmc1223-sudo/hermes-agent-fluxmem
that referenced
this pull request
Jul 8, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…ousResearch#54465) (NousResearch#55544) The auxiliary OpenAI clients were built without overriding the SDK's default max_retries=2, so every aux call silently made up to 3 attempts against a slow/hung endpoint — a 120s timeout could stall ~360s before Hermes saw a single failure. On the critical compression preflight path, Hermes then added its own same-provider timeout retry on top, roughly doubling the user-visible stall again before fallback. - Build both the sync (_create_openai_client) and async (_to_async_client) aux clients with max_retries=0 (setdefault, so explicit callers still override). Hermes already owns retry + provider/model fallback policy. - For task == compression, skip the same-provider transient retry on a full-budget timeout and fall straight through to fallback. Fast blips (streaming-close, 5xx) still retry, since those are cheap. - Add _is_timeout_error to distinguish a full-budget timeout from a fast connection drop. Addresses the retry-multiplication root cause of NousResearch#54465 (the resume-wedge persistence half landed in NousResearch#55499).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resumed oversized CLI sessions no longer wedge when the auxiliary compression endpoint times out (#54465). The same-session compression-failure cooldown now survives a process restart (it was in-memory only), and the compression lock lease is refreshed while a long compression call is still in flight (the 300s TTL was shorter than a single ~361s aux call).
Salvaged from #54525 by @rodboev — their commits are cherry-picked unchanged to preserve authorship; one follow-up commit by me hardens the lease refresher and adds regression tests.
Changes
@rodboev's work (cherry-picked, 6 commits):
hermes_state.py: session-scopedcompression_failure_cooldown_until/compression_failure_errorcolumns (declarative reconciliation — no SCHEMA_VERSION bump) +record/get/clear_compression_failure_cooldownand owner-checkedrefresh_compression_lockhelpers.agent/context_compressor.py: bind resumed session state into the built-in compressor; write through same-session cooldown; preserve manual/compressforce-bypass.agent/agent_init.py,run_agent.py: rebind the compressor onto the active session row on fresh / reset-only (/new,/resume,/branch) switches.agent/turn_context.py: skip automatic preflight compression while a same-session cooldown is live.agent/conversation_compression.py: keep the lock lease alive (background refresher) until release; release on every early exit path.Follow-up hardening (1 commit, by me):
cap = int(ttl / refresh_interval), floor 1), so the give-up window is genuinely bounded by the TTL and a transient blip recovers on the next tick.except Exception: passarms in the cooldown persist/clear helpers with debug logging, for parity with theirsqlite3.Errorsiblings (a non-sqlite bug was previously invisible).join(timeout=1.0)quiesce bound instop().break-on-first-failure).Validation
tests/test_hermes_state.py tests/agent/test_context_compressor.py tests/agent/test_context_engine_host_contract.py tests/agent/test_turn_context.py tests/agent/test_compression_concurrent_fork.pySessionDB, cross-process): cooldown recorded in one process is hydrated by a fresh process → preflight skipped on resume.Closes #54465