fix: stop gateway memory leak on cached agent and session expiry

[NeroNarada]

Summary

Fixes a long-lived gateway memory leak introduced by cache/session lifecycle paths that never fully release state on agent eviction or expiry.

Changes

• Ensure _evict_cached_agent() actually detaches and cleans the evicted agent on /new, /model, and related reset paths by invoking the existing soft-eviction cleanup path asynchronously.
• Extend soft cleanup to clear _session_messages, releasing potentially large per-session histories.
• Clear stale per-session dictionaries when idle session expiry finalizes a session:
  • _session_model_overrides
  • _session_reasoning_overrides
  • _pending_approvals
  • _update_prompt_pending
• Add a bounded policy to quiet-mode tool-definition cache in model_tools.py:
  • cap at 8 entries
  • reset cache when over capacity before inserting a new key
• Add regressions covering
  • evicted cached agent soft cleanup and message clearing,
  • session-expiry state-map cleanup,
  • tool-definition cache cap behavior.

Issue

• Fixes fix: gateway memory leak — _evict_cached_agent drops agents without cleanup, _session_messages never cleared #25315

[liuhao1024]

Overall: Solid multi-area fix — the session-scoped state cleanup, agent cache soft-release, and localhost proxy bypass are all well-targeted. One concern:

_tool_defs_cache clear-on-cap strategy risks perpetual thrashing

In model_tools.py:

if len(_tool_defs_cache) >= _TOOL_DEFS_CACHE_MAX_ENTRIES and cache_key not in _tool_defs_cache:
    _tool_defs_cache.clear()

When the gateway serves >8 distinct toolset configurations (which is plausible when different sessions have different enabled_toolsets/disabled_toolsets), the 9th distinct config clears the entire cache — not just the oldest entry. Every subsequent call with a config that was previously cached will miss, and the single freshly-inserted entry gets evicted on the next distinct config.

In the worst case (e.g. a gateway serving 10+ platform sessions each with unique toolset combos), this degrades to zero cache effectiveness: every call recomputes tool definitions, which is worse than having no cache at all.

Suggestion: Use an LRU-eviction approach (drop the oldest entry) or a random single-key eviction instead of clear():

if len(_tool_defs_cache) >= _TOOL_DEFS_CACHE_MAX_ENTRIES and cache_key not in _tool_defs_cache:
    # Evict the oldest entry instead of clearing everything
    oldest_key = next(iter(_tool_defs_cache))
    del _tool_defs_cache[oldest_key]

Or even simpler with collections.OrderedDict / dict (Python 3.7+ preserves insertion order):

if len(_tool_defs_cache) >= _TOOL_DEFS_CACHE_MAX_ENTRIES and cache_key not in _tool_defs_cache:
    _tool_defs_cache.pop(next(iter(_tool_defs_cache)), None)

This keeps the remaining 7 cached entries warm, which matters in a multi-session gateway.