fix(compression): tripped sessions recover across gateway agent rebuilds — durable recovery deadline column (salvage #100185) - #101192
Merged
Conversation
…y agent rebuilds cannot block a session forever The #14694 recovery clock (`_anti_thrash_recovery_deadline`) was a process-local `time.monotonic()` value zeroed in `bind_session_state()`. The gateway rebuilds the AIAgent (and its ContextCompressor) on every cache eviction, so each fresh compressor bound to a durably tripped session row (#69872) re-armed a full 300s window and the half-open probe never fired — a long messaging conversation above the threshold stayed blocked permanently. Persist the deadline as a wall-clock epoch in a new `sessions.compression_recovery_deadline REAL` column (declarative column reconciliation; SCHEMA_VERSION 26 -> 27) with `SessionDB.get/set_compression_recovery_deadline`. The compressor loads it in `bind_session_state()` and writes it on change only via `_set_anti_thrash_recovery_deadline()`. A fresh compressor with no stored deadline still starts a full window blocked (#54923 restart contract); one that loads an armed deadline resumes that window. Backward clock jumps are bounded to one window. The 300s window is unchanged. Minimal salvage of #100185 (the probe-lease/fencing state machine and model_config-blob storage were not carried). Refs #100185 Co-authored-by: Komzpa <me@komzpa.net>
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Blocked (anti-thrash-tripped) sessions now earn their 300s recovery probe across gateway agent rebuilds: the recovery deadline is persisted on the session row as a wall-clock epoch instead of living in a process-local
time.monotonic()value that every freshContextCompressorreset — so a long messaging conversation above the compression threshold no longer stays blocked forever.Minimal salvage of #100185 by @Komzpa (credited via
Co-authored-by). Per Teknium's decision, only the durable-deadline piece is carried: properSessionDBcolumn (not themodel_configJSON blob), 300s window unchanged, no probe-lease/fencing state machine, no rerouted breaker test.Changes
hermes_state_common.py: newsessions.compression_recovery_deadline REALcolumn inSCHEMA_SQL(declarative column reconciliation adds it to existing DBs);SCHEMA_VERSION26 → 27.hermes_state.py:SessionDB.get_compression_recovery_deadline()/set_compression_recovery_deadline()(0/None = disarmed), mirroring thecompression_ineffective_countaccessors.agent/context_compressor.py:_ANTI_THRASHclock now usestime.time()(wall clock) so it is meaningful across processes._load_anti_thrash_recovery_deadline()inbind_session_state();_set_anti_thrash_recovery_deadline()persists on change only (arm / disarm / probe-granted).now + 300s, so a wait can never exceed one window.tests/agent/test_compression_anti_thrash_recovery.py: existing tests moved to the wall clock; newTestDurableDeadline(fresh compressors resume the same window, stay blocked inside it, backward-jump bound, disarm-on-clear, SessionDB round trip).tests/state/test_session_git_metadata_generation.py:== SCHEMA_VERSION == 26change-detector literal →== SCHEMA_VERSION.Validation
tests/agent/test_compression_anti_thrash_recovery.py+..._anti_thrash_persistence.py+test_compaction_anti_thrash.py+..._split_failure_cooldown.py+tests/state/test_session_git_metadata_generation.pytests/agent/test_compression_concurrent_fork.py+tests/test_hermes_state.py+test_context_compressor*.py(3 files)TestFTS5Search::test_search_projection_skips_context_enrichment_queries— fails identically on untouchedorigin/main, unrelated)ruff checkon touched filesscripts/audit_pr_attribution.py --fixLive repro: real-import harness (
SessionDBon a tempHERMES_HOME,set_compression_ineffective_count(sid, 2), three FRESH compressors bound to the same row with the wall clock advanced +20 min and +40 min between them) — before:B (fresh, +20min) blocked: True,C (fresh, +40min) blocked: True, db count stays 2 →REPRO_FIRES: fresh compressors never probe; deadline restarts per instance; after:B (fresh, +20min) blocked: False db count: 1,C ... blocked: False→CLEAN.Refs #100185 — salvaged from @Komzpa's PR (authorship credited via
Co-authored-by); the broader lease/fencing design there is intentionally not carried.Infographic