[Bug]: gateway cross-process agent-cache invalidation performs cleanup while holding _agent_cache_lock, stalling the event loop and blocking Discord heartbeats

[eyeonall]

Bug Description

A recent gateway regression appears to perform expensive cached-agent cleanup while holding self._agent_cache_lock in the cross-process session-write invalidation path. In a live Discord gateway, this can stall the asyncio loop long enough to trigger repeated:

discord.gateway: Shard ID None heartbeat blocked for more than N seconds

In the affected runtime, the traceback consistently points at the session-expiry watcher trying to acquire the same cache lock:

• gateway/run.py::_session_expiry_watcher
• gateway/run.py::_sweep_idle_cached_agents
• blocked at with _lock:

The likely bad interaction is:

1. A turn detects a cross-process transcript change via message_count mismatch.
2. The invalidation path acquires _agent_cache_lock.
3. It pops the cached agent and calls self._cleanup_agent_resources(_ev_agent) while still inside the lock.
4. Cleanup can be slow/blocking (shutdown_memory_provider(), close(), async-client cleanup, etc.).
5. The session-expiry watcher later runs on the gateway loop and blocks trying to acquire the same lock in _sweep_idle_cached_agents().
6. The event loop stalls and Discord heartbeat warnings accumulate.

This looks like a regression in the cross-process agent-cache coherence work from #45966.

Steps to Reproduce

I do not yet have a fully deterministic minimal repro, but the live trigger pattern is:

1. Run a long-lived Discord gateway.
2. Use a session that may receive transcript updates from another process or path that shares the same SessionDB / session state.
3. Trigger a follow-up turn where the gateway detects:
   • message_count changed (...), possible cross-process write
4. While the invalidated cached agent is being cleaned up, let session-expiry maintenance run.
5. Observe Discord heartbeat-blocked warnings and multi-minute loop stalls.

The repro may be easiest when the evicted agent has non-trivial resources to clean up.

Expected Behavior

Cross-process cache invalidation should remove the stale cache entry quickly and release the cache lock immediately.

Any potentially expensive cleanup should happen after the lock is released, ideally using the same pattern already used by _evict_cached_agent() / idle eviction / cap enforcement.

The gateway event loop should remain responsive and Discord heartbeats should not be blocked by cache-maintenance work.

Actual Behavior

The gateway event loop can stall for minutes after a cross-process cache invalidation event, causing repeated Discord heartbeat-blocked warnings and very delayed responses.

Affected Component

Gateway (Telegram/Discord/Slack/WhatsApp)

Messaging Platform (if gateway-related)

Discord

Debug Report

Environment:
- Profile: `karl`
- Gateway service: `hermes-gateway-karl.service`
- Main PID during observation: `7726`

Observed journal pattern:


WARNING discord.gateway: Shard ID None heartbeat blocked for more than 270 seconds.
...
File ".../gateway/run.py", line 5875, in _session_expiry_watcher
  _idle_evicted = self._sweep_idle_cached_agents()
File ".../gateway/run.py", line 13396, in _sweep_idle_cached_agents
  with _lock:


The warnings repeated at 280s, 290s, 300s, 310s, 320s, etc.

In the profile gateway log, the stall lines up with a cross-process invalidation event:


2026-06-23 11:22:25 inbound message received
2026-06-23 11:23:43 Agent cache invalidated for session ... message_count changed (51 -> 70), possible cross-process write
2026-06-23 11:29:53 Discord reconnects
2026-06-23 11:31:11 response ready ... time=525.5s


That timing strongly suggests the invalidation/cleanup path stalled the loop long enough to wedge heartbeats and delay the user-visible response.

## Why this looks real (not just a generic Discord/network issue)

- The traceback consistently points at Hermes code waiting on `_agent_cache_lock`, not external network I/O.
- The stall lines up with `Agent cache invalidated ... possible cross-process write` in the same session.
- Other gateway code already treats cleanup-under-lock as unsafe and goes out of its way to avoid it.
- The exact cleanup call appears to have been introduced by the #45966 fix and is still present on current `origin/main`.

Operating System

Fedora 44

Python Version

3.13.13

Hermes Version

Observed on local checkout:

• local HEAD: 73cd8622f
• origin/main at inspection time: 5ecf3bf0e

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

Upstream History / Suspected Introducing Change

The problematic path appears to have been introduced here:

• PR fix(gateway): invalidate agent cache on cross-process session writes (#45966) #46095 — fix(gateway): invalidate agent cache on cross-process session writes (#45966)
• PR fix(gateway): cross-process agent-cache coherence (#45966) + preserve prompt caching #46237 — fix(gateway): cross-process agent-cache coherence (#45966) + preserve prompt caching

The follow-up in #46237 fixes same-process prompt-cache churn by re-baselining message_count, but it does not appear to change the cleanup-under-lock behavior.

I checked current origin/main and the exact pattern still exists there.

Current Source Evidence

On current origin/main, the cross-process invalidation path still does:

with _cache_lock:
    ...
    evicted = self._agent_cache.pop(session_key, None)
    _ev_agent = evicted[0] if isinstance(evicted, tuple) and evicted else None
    if _ev_agent and _ev_agent is not _AGENT_PENDING_SENTINEL:
        self._cleanup_agent_resources(_ev_agent)

By contrast, other cache-eviction paths already avoid this pattern:

• _evict_cached_agent() explicitly says cleanup runs on a daemon thread so the code does not block while holding _agent_cache_lock
• _sweep_idle_cached_agents() collects entries under the lock, drops them from the cache, then releases resources outside the lock

So this cross-process invalidation path seems inconsistent with the gateway’s own existing lock-discipline elsewhere.

Why this looks real (not just a generic Discord/network issue)

• The traceback consistently points at Hermes code waiting on _agent_cache_lock, not external network I/O.
• The stall lines up with Agent cache invalidated ... possible cross-process write in the same session.
• Other gateway code already treats cleanup-under-lock as unsafe and goes out of its way to avoid it.
• The exact cleanup call appears to have been introduced by the Gateway replies with stale context when another process appends to the same session (cross-process agent-cache split-brain) #45966 fix and is still present on current origin/main.

Proposed Fix (optional)

Hermes wanted me to submit a PR however this is still new so I had it propose the fix.

Make the cross-process invalidation path match the existing lock discipline used elsewhere:

• hold _agent_cache_lock only long enough to inspect cache state and pop the stale entry
• store _ev_agent in a local variable
• release the lock
• perform _cleanup_agent_resources(_ev_agent) (or a soft-release variant, depending on desired semantics) outside the lock

A simple first step would be to refactor that branch to mirror _sweep_idle_cached_agents() / _evict_cached_agent().

Additional Notes

Related but distinct heartbeat-blocking work already exists in the repo for other synchronous event-loop blockers, e.g. handoff watcher SQLite polling (#40695 / #40974 / #43504) and other gateway loop blockers. This issue looks like the same class of problem, but on the agent-cache invalidation path.

Version / Repo Context

Observed on local checkout:

• local HEAD: 73cd8622f
• origin/main at inspection time: 5ecf3bf0e

At inspection time, the exact cleanup-under-lock line was still present on origin/main.

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR

[alt-glitch]

This was generated by AI during triage.

Related to #45966 (cross-process agent-cache split-brain) and its fix PR #46095, which introduced the cross-process message_count-mismatch invalidation path. Confirmed in main: gateway/run.py calls the full-teardown _cleanup_agent_resources(_ev_agent) inside with _agent_cache_lock: on the invalidation branch, while _sweep_idle_cached_agents() (driven by _session_expiry_watcher) contends for the same lock — a slow teardown (memory-provider shutdown, client close) under the lock can stall the asyncio loop and trip Discord heartbeat-blocked warnings. Treated as a regression of #46095, not a duplicate.