Skip to content

fix(gateway): clean up _running_agents_ts on agent removal paths - #8119

Closed
chinadbo wants to merge 3 commits into
NousResearch:mainfrom
chinadbo:fix/running-agents-ts-orphan-cleanup
Closed

chinadbo wants to merge 3 commits into
NousResearch:mainfrom
chinadbo:fix/running-agents-ts-orphan-cleanup

Conversation

@chinadbo

Copy link
Copy Markdown
Contributor

Summary

  • _running_agents_ts entries were orphaned when agents were removed from _running_agents via /stop, /new, /resume, shutdown, or hard-stop of pending agents
  • Added _running_agents_ts.pop(key, None) after every del _running_agents[key] (6 sites) and _running_agents_ts.clear() after _running_agents.clear() in shutdown (1 site)
  • Existing code already had the correct pattern in 3 places (stale eviction line 2449, finally-block cleanup line 2864, notify_long_running line 8376) — this PR makes all deletion sites consistent

Test plan

  • test_stop_command_cleans_ts — stop path cleans up timestamp
  • test_new_command_cleans_ts — new/reset path cleans up timestamp
  • test_resume_command_cleans_ts — resume path cleans up timestamp
  • test_shutdown_clears_ts — shutdown clears timestamps
  • test_all_del_sites_have_ts_pop — source-level scan ensures no del without pop within 5 lines
  • test_clear_site_also_clears_ts — source-level scan ensures no .clear() without ts .clear() within 5 lines
  • All existing runner tests pass

@chinadbo
chinadbo force-pushed the fix/running-agents-ts-orphan-cleanup branch from 74691e7 to 8e6a4c8 Compare April 27, 2026 07:29
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery labels Apr 27, 2026
@chinadbo
chinadbo force-pushed the fix/running-agents-ts-orphan-cleanup branch from fa1e8d9 to a228713 Compare April 27, 2026 13:18
chinadbo and others added 3 commits April 28, 2026 10:03
When agents were removed from _running_agents (via /stop, /new, /resume,
shutdown, or hard-stop of pending agents), the corresponding entry in
_running_agents_ts was not cleaned up, causing orphaned timestamps that
accumulate as a memory leak and can corrupt stale-timeout checks.

Add _running_agents_ts.pop() after every del _running_agents[key] and
_running_agents_ts.clear() after _running_agents.clear() in shutdown.
- Replace tests that simulated cleanup inline with real _release_running_agent_state calls
- Remove vacuous test_all_del_sites_have_ts_pop (trivially passes, redundant)
- Add test_release_also_cleans_busy_ack_ts asserting all three dicts cleared
- Remove unused AsyncMock import
The previous test called .clear() on the dicts in the test body itself,
trivially passing without testing any production code path.  Replace it
with a source-level assertion that the shutdown block in gateway/run.py
contains _running_agents_ts.clear() after _running_agents.clear() in
the expected order.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@chinadbo
chinadbo force-pushed the fix/running-agents-ts-orphan-cleanup branch from a228713 to 43ae049 Compare April 28, 2026 02:04
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for this careful analysis and the well-structured test plan, @chinadbo!

This automated hermes-sweeper review found that the exact bug you identified was already resolved on main by PR #11800 (commit 31e727647, shipped in v2026.4.23, Apr 17 2026), which was merged a few days after this PR was opened.

That fix took the same approach you identified — but went a step further:

  • _release_running_agent_state(session_key) (gateway/run.py:11102) — a single helper that atomically pops _running_agents, _running_agents_ts, and _busy_ack_ts in lockstep.
  • All 8 bare del self._running_agents[key] call sites were replaced with the helper; rg "del self._running_agents\[" gateway/run.py now returns only the docstring reference inside the helper.
  • The shutdown path also calls _running_agents.clear() + _running_agents_ts.clear() via the helper.
  • tests/gateway/test_session_state_cleanup.py (added by fix(gateway): consolidate per-session cleanup; close SessionDB on shutdown #11800) covers stop/new/resume/shutdown cleanup plus a source-level regression guard that fails if any future contributor reintroduces a bare del outside the helper.

Your contribution helped confirm this was a real, worth-fixing leak — the eventual fix just landed independently via the same diagnosis path you took. Closing as implemented on main.

@teknium1 teknium1 closed this Jun 10, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants