Skip to content

fix(web_server): reap finished action subprocesses to prevent zombie accumulation - #38045

Closed
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/web-server-zombie-action-procs
Closed

fix(web_server): reap finished action subprocesses to prevent zombie accumulation#38045
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/web-server-zombie-action-procs

Conversation

@liuhao1024

Copy link
Copy Markdown
Contributor

What does this PR do?

Reaps finished dashboard action subprocesses in get_action_status() to prevent zombie process accumulation. Previously, proc.poll() was called to check liveness but proc.wait() was never issued, leaving every completed action as a <defunct> zombie for the lifetime of the web server.

Related Issue

Fixes #38032

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/web_server.py: After proc.poll() returns a non-None exit code, call proc.wait(timeout=1) to reap the child process and .pop() the handle from _ACTION_PROCS. This prevents zombie accumulation for all dashboard actions (gateway-restart, hermes-update, etc.).
  • tests/hermes_cli/test_web_server.py: Added test_finished_action_proc_is_reaped_and_removed — injects a mock FinishedProc into _ACTION_PROCS, calls the status endpoint, and asserts that proc.wait() was called and the entry was removed from the dict.

How to Test

  1. Run the new test: pytest tests/hermes_cli/test_web_server.py::TestWebServerEndpoints::test_finished_action_proc_is_reaped_and_removed -xvs
  2. Run related action tests: pytest tests/hermes_cli/test_web_server.py -k "action" -xvs
  3. Verify no regressions: pytest tests/hermes_cli/test_web_server.py -x

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/hermes_cli/test_web_server.py -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Code Intelligence

  • Analyzed: hermes_cli/web_server.py:get_action_status() (caller: FastAPI route handler, flows: dashboard action status polling)
  • Blast radius: LOW — localized to action status endpoint; no behavioral change for running processes
  • Related patterns: _ACTION_RESULTS dict already handles completed non-spawned actions; this fix extends the same cleanup to spawned subprocesses

…accumulation

get_action_status() calls proc.poll() to check if a dashboard action
has finished, but never calls proc.wait() afterward.  On POSIX systems
the kernel retains the process table entry until a blocking waitpid()
is issued, so every completed action remains as a zombie for the
lifetime of the web server.

After poll() returns a non-None exit code, call proc.wait(timeout=1)
to reap the child and remove the handle from _ACTION_PROCS.

Fixes NousResearch#38032
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels Jun 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Competing PR: #38040 also fixes #38032 and is broader in scope — it reaps the dashboard action zombies (same proc.wait() fix as here) plus the meet_bot.py pcm_pump leak and additional teardown in plugins/platforms/discord/adapter.py and tui_gateway/server.py. This PR is the minimal/focused variant (web_server.py only, with a dedicated reap test). Maintainers should pick one; see also #27073 / #35626 in the broader zombie-reap family.

@teknium1

teknium1 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Superseded by #38049 (commit 78e2101), now merged to main — closes #38032. Your fix was the same approach (reap finished children via proc.wait() on poll); thanks for the clean, well-scoped patch. The merged version additionally migrates the exit code into _ACTION_RESULTS so repeat status polls stay consistent.

@teknium1 teknium1 closed this Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: web_server.py: dashboard action subprocesses become zombies — _ACTION_PROCS uses .poll() but never .wait()

3 participants