fix(gateway): bound the agent cache by memory, not just count and age (#80764, salvage of #80795) - #81127
Merged
kshitijk4poor merged 3 commits intoAug 7, 2026
Conversation
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and neither bound knows how many bytes it holds. Each cached agent pins _session_messages -- the full transcript including tool output, tens of MB on a session with 100+ tool calls -- so a gateway serving many chats keeps every warm transcript resident: agents that took a turn inside the TTL are never idle-swept, and the idle sweep additionally defers finalizable sessions until they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer flush inside systemd's stop timeout. Add the missing bound. Each session-expiry watcher tick compares the process's anonymous RSS against a budget and, when over, sheds LRU agents through the same soft-eviction path the cap enforcer uses, then runs malloc_trim so the freed arenas actually return to the OS. Evicted sessions rebuild their transcript from the persisted session on the next turn. Three classes of session are never shed: agents mid-turn, the most recently used ones, and any session whose transcript has not finished reaching disk (_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS write-corruption guard reacts to when it preserves live history). memory_high_mb defaults to "auto", deriving the budget from the cgroup limit the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected without a second number to keep in sync. The two existing bounds become configurable alongside it under agent.agent_cache. protect_recent is clamped to half the cache: a couple of sessions can exhaust the budget on their own, and a fixed MRU guard would then protect everything and leave the gateway climbing with nothing it would shed. Fixes NousResearch#80764
Record why the cache needs a third bound and what the pressure pass will and will not shed, so an operator tuning agent.agent_cache knows which knob to reach for. Adds the config keys to the session-lifecycle appendix and a user guide section covering the "auto" cgroup-derived budget.
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
4 tasks
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The gateway's per-session agent cache gains the missing third bound — memory. Once the process's anonymous RSS crosses a budget derived from the cgroup limit it actually runs under, least-recently-used transcripts are shed through the existing soft-eviction path and the heap is returned to the OS, so a busy multi-platform gateway stops climbing to the cgroup throttle and dying to SIGKILL every 4-7h (#80764).
Salvage of #80795 by @HexLab98 (both commits cherry-picked, authorship preserved). Closes #80795. Fixes #80764. Supersedes #47848 (idle TTL + max_size become configurable under
agent.agent_cache).Context
Before: the cache was bounded by entry count (128) and idle time (1h) — both blind to bytes. Each cached agent pins
_session_messages(full transcript incl. tool output, tens of MB on 100+-tool-call sessions); agents active within the TTL are never swept, so RSS grows untilmemory.highthrottling stalls the SIGTERM flush past systemd's stop timeout. After: a 300s watcher tick compares anon-RSS to the budget and sheds LRU sessions (never mid-turn agents, never the MRUprotect_recent, never sessions whose transcript hasn't reached disk); evicted sessions rebuild from the persisted session next turn with the stored system prompt reused verbatim (prompt-cache byte-stability preserved).Changes
gateway/agent_cache_pressure.py(config resolution, cgroup budget, RSS reading, eviction planner),GatewayRunner._sweep_agent_cache_under_pressure(), configurablemax_size/idle_ttl_secs, 36 tests (persistence guard exercised against a realAIAgent+SessionDB), docs. Conflict with b3e9e91 inconfig_defaults.pyresolved both-sides-keep (lease key retained).trim_memory— the batch thread held every evicted agent in its local list whilegc.collect+malloc_trimran, so the in-pass trim freed ~nothing and the next tick over-evicted an extra batch of warm prompt caches per cycle._db_flush_scan_prefixon soft release — a shallow copy of the flushed transcript that pinned the multi-MB content strings on exactly the agents the valve targets (pressure-evictable ⇒ flushed ⇒ populated).resolve_agent_cache_bounds({})("auto") instead of a permanently-cached disabled valve.protect_recent: false(YAML bool,False == 0) keeps the default MRU protection.HERMES_HOME) from mid-turn agents, so the one deployment class where the valve correctly refuses to shed is diagnosable._cgroup_limit_byteschecks the process's own cgroup (reusinggateway.cgroup_cleanup._own_cgroup_path) before the root files, so systemdMemoryHigh=/MemoryMax=unit limits are detected.Validation
Concurrency traced during review: snapshot-before-lock window is the same class as the existing sweeps and is closed in practice by the under-lock
_last_flushed_db_idx=0reset + MRU protection; pop-then-release is identical in semantics to the cap-enforcer path (release_clientstouches no shared per-session resources);trim_memoryno-ops gracefully on macOS.Provenance: contributor's design (budget derivation, planner, guard, soft-path reuse, all 36 tests) kept intact; review findings applied on top as a separate commit. Nothing dropped.