Skip to content

fix(agent): reap browser/VM task resources on aborted turns - #39897

Closed
tim404x wants to merge 1 commit into
NousResearch:mainfrom
tim404x:fix/browser-task-cleanup-leak
Closed

fix(agent): reap browser/VM task resources on aborted turns#39897
tim404x wants to merge 1 commit into
NousResearch:mainfrom
tim404x:fix/browser-task-cleanup-leak

Conversation

@tim404x

@tim404x tim404x commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

The gateway leaked memory slowly under real use: every conversation turn that aborted with an exception or was interrupted left its per-task resources behind. The agent-browser daemon and its Chromium/Xvfb process tree (plus any sandbox VM) would orphan to init and accumulate.

Two root causes, both fixed:

  1. run_conversation only cleaned up task resources on its normal return paths. On the exception / interrupt path it returned without reaping. This splits the function into a thin public run_conversation() wrapper and _run_conversation_impl(): the wrapper runs _cleanup_task_resources() in a finally that fires ONLY when the impl did not complete normally, so the success path is unchanged (no double cleanup, no timing change) while aborts are always reaped. Cleanup is idempotent and persistence-aware, so it is safe to run on abort.

  2. _terminate_host_pid only sent SIGTERM to the process tree snapshot taken at call time. Processes that ignore SIGTERM (chrome_crashpad_handler, renderers under load) survived, and children reparented to init as their parents exited fell off children(recursive=True) entirely. It now snapshots the whole tree up front, SIGTERMs it (letting a live browser flush its already-persisted cookies), waits, then SIGKILLs anything still alive and reaps zombies.

Tests: tests/tools/test_run_conversation_abort_cleanup.py covers the wrapper (normal return does not double-reap; exception and interrupt both reap with the correct task id; a generated id is reused for the reap; a failing reaper does not mask the original turn error). Regression-validated. The existing browser-orphan-reaper suite (18 tests) still passes.

What does this PR do?

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

How to Test

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform:

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

  • This skill is broadly useful to most users (if bundled) — see Contributing Guide
  • SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
  • No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
  • I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/browser Browser automation (CDP, Playwright) labels Jun 5, 2026
The gateway leaked memory slowly under real use: every conversation turn
that aborted with an exception or was interrupted left its per-task
resources behind. The agent-browser daemon and its Chromium/Xvfb process
tree (plus any sandbox VM) would orphan to init and accumulate.

Two root causes, both fixed:

1. run_conversation only cleaned up task resources on its normal return
   paths. On the exception / interrupt path it returned without reaping.
   This splits the function into a thin public run_conversation() wrapper
   and _run_conversation_impl(): the wrapper runs _cleanup_task_resources()
   in a finally that fires ONLY when the impl did not complete normally, so
   the success path is unchanged (no double cleanup, no timing change) while
   aborts are always reaped. Cleanup is idempotent and persistence-aware, so
   it is safe to run on abort.

2. _terminate_host_pid only sent SIGTERM to the process tree snapshot taken
   at call time. Processes that ignore SIGTERM (chrome_crashpad_handler,
   renderers under load) survived, and children reparented to init as their
   parents exited fell off children(recursive=True) entirely. It now
   snapshots the whole tree up front, SIGTERMs it (letting a live browser
   flush its already-persisted cookies), waits, then SIGKILLs anything still
   alive and reaps zombies.

Tests: tests/tools/test_run_conversation_abort_cleanup.py covers the wrapper
(normal return does not double-reap; exception and interrupt both reap with
the correct task id; a generated id is reused for the reap; a failing reaper
does not mask the original turn error). Regression-validated. The existing
browser-orphan-reaper suite (18 tests) still passes.
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused resource-cleanup work. This is an automated hermes-sweeper review; the requested behavior is now implemented on current main with a stronger tree-reaping implementation.

  • agent/conversation_loop.py:647-653, :2300-2323, and :5275-5351 funnel interrupt and ordinary exception paths into finalize_turn; agent/turn_finalizer.py:180-185 independently reaps _cleanup_task_resources(effective_task_id).
  • tools/process_registry.py:620-665 snapshots the complete live tree, SIGTERMs it, then re-probes every captured process and SIGKILLs survivors. This is stronger than relying on wait_procs' alive partition.
  • Commit 8cfcbd327dfc65dbc073d0ba002dbff7a61f7713 added the full-tree regression at tests/tools/test_process_registry.py:1899-1961; the implementation shipped in v2026.7.1.
  • The later cross-reference, fix(browser): kill orphaned Chromium processes when daemon is dead #60152, concerns the separate case where the daemon is already dead and Chromium has already been reparented; it does not require this PR's older implementation.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/browser Browser automation (CDP, Playwright) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants