fix(gateway): bound the agent cache by memory, not just count and age - #80795
Closed
HexLab98 wants to merge 2 commits into
Closed
fix(gateway): bound the agent cache by memory, not just count and age#80795HexLab98 wants to merge 2 commits into
HexLab98 wants to merge 2 commits into
Conversation
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and neither bound knows how many bytes it holds. Each cached agent pins _session_messages -- the full transcript including tool output, tens of MB on a session with 100+ tool calls -- so a gateway serving many chats keeps every warm transcript resident: agents that took a turn inside the TTL are never idle-swept, and the idle sweep additionally defers finalizable sessions until they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer flush inside systemd's stop timeout. Add the missing bound. Each session-expiry watcher tick compares the process's anonymous RSS against a budget and, when over, sheds LRU agents through the same soft-eviction path the cap enforcer uses, then runs malloc_trim so the freed arenas actually return to the OS. Evicted sessions rebuild their transcript from the persisted session on the next turn. Three classes of session are never shed: agents mid-turn, the most recently used ones, and any session whose transcript has not finished reaching disk (_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS write-corruption guard reacts to when it preserves live history). memory_high_mb defaults to "auto", deriving the budget from the cgroup limit the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected without a second number to keep in sync. The two existing bounds become configurable alongside it under agent.agent_cache. protect_recent is clamped to half the cache: a couple of sessions can exhaust the budget on their own, and a fixed MRU guard would then protect everything and leave the gateway climbing with nothing it would shed. Fixes NousResearch#80764
Record why the cache needs a third bound and what the pressure pass will and will not shed, so an operator tuning agent.agent_cache knows which knob to reach for. Adds the config keys to the session-lifecycle appendix and a user guide section covering the "auto" cgroup-derived budget.
alt-glitch
added
type/perf
Contributor
|
This was generated by AI during triage. Summary: Problems:
Solution: Checked against |
kshitijk4poor
added a commit
that referenced
this pull request
Aug 7, 2026
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
Collaborator
|
Merged via #81127 — both your commits were cherry-picked with authorship preserved, so this lands in main's history under your name. This was a genuinely well-built fix: the cgroup-derived auto budget, the eviction planner with the half-cache clamp, and especially the persistence guard tested against a real AIAgent + SessionDB all survived review intact. On top of your commits we added a follow-up (review findings, separate commit): draining the eviction batch before |
ma1138569845
pushed a commit
to ma1138569845/dechnicAuditor-agent
that referenced
this pull request
Aug 10, 2026
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
blut-agent
pushed a commit
to blut-agent/hermes-agent-fork
that referenced
this pull request
Aug 11, 2026
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #80764 — gateway RSS grows unbounded under sustained multi-session load until the cgroup throttles and systemd SIGKILLs the process.
The per-session agent cache is bounded by entry count (
_AGENT_CACHE_MAX_SIZE = 128) and by idle time (_AGENT_CACHE_IDLE_TTL_SECS = 3600). Neither knows how many bytes it holds. Each cached agent pins_session_messages— the full transcript including tool output, tens of MB on a session with 100+ tool calls — so a busy gateway keeps every warm transcript resident: agents that took a turn inside the TTL are never idle-swept, and_sweep_idle_cached_agentsadditionally defers finalizable sessions until they expire. The 06-08 fix (#41974) released the LLM clients on eviction but left the transcripts.This adds the missing third bound.
gateway/agent_cache_pressure.py— config resolution, anonymous-RSS reading, budget derivation, and the eviction planner. Kept out ofrun.pyso the policy is testable without a gateway.GatewayRunner._sweep_agent_cache_under_pressure()runs on the existing session-expiry watcher tick (no new thread). Over budget, it sheds LRU agents through the same soft path_enforce_agent_cache_capuses (_commit_then_release_soft, so a finalizable session still gets itson_session_endextraction), then runsmalloc_trim— without that glibc keeps the freed arenas and the cgroup never sees the drop.protect_recentmost-recently-used sessions, and any session whose transcript has not finished reaching disk. The last one compares_last_flushed_db_idxagainstlen(_session_messages)— the same divergence the FTS write-corruption guard reacts to atrun.py:5113when it preserves live history over a lagging transcript.agent.agent_cache, superseding feat(gateway): make agent cache idle TTL configurable #47848.memory_high_mb: autoderives the budget from the cgroup limit the gateway actually runs under (memory.high, thenmemory.max, then cgroup v1), falling back to total RAM when uncapped, so aMemoryHigh/MemoryMaxon the unit is respected without a second number to keep in sync. That is what makes this work out of the box on the deployments where the leak bites.Two notes on the design:
protect_recentis clamped to half the cache. A fixed MRU guard would protect the entire cache in the [Bug]: TUI Gateway progressive RSS leak — 8 concurrent sessions, 7.4 GB tui_gateway RSS #62743 shape (7.4 GB across 8 sessions) and leave the gateway climbing toward the OOM killer with nothing it would shed.ephemeral_pin/vc_last, matching the cap enforcer and idle sweep: the session continues, so the rebuilt agent should render the same session-context bytes. Only true boundaries (_evict_cached_agent) reset them.Test plan
scripts/run_tests.sh tests/gateway/test_agent_cache_pressure.py— 36 new tests. The persistence guard is exercised against a realAIAgent+ realSessionDBthrough the actual_flush_messages_to_session_db, not mocks: an unflushed transcript blocks eviction, and a successful flush unblocks it.scripts/run_tests.sh tests/gateway/— same failure set as the pre-change baseline on this machine (test_teams,test_whatsapp_bridge_pidfile, git/pty-dependent files), verified by stashing.scripts/run_tests.sh tests/gateway/test_agent_cache.pyand the other eviction-path suites (test_session_boundary_hooks,test_shutdown_cache_cleanup,test_model_command_expensive_confirm,test_compression_deferred_soft_result,test_10710_auto_reset_evicts_cached_agent) — green.scripts/run_tests.sh tests/hermes_cli/test_config.py tests/hermes_cli/test_mem_trim.py— green.Behaviour is unchanged when memory is fine: below the budget the pass returns immediately, and with an empty cache it does not run at all. The reporter offered 48h of
Pss_Anontelemetry against a patch; this is the shape they proposed, so that verification would apply directly.