Skip to content

fix(gateway): remove stale foreign pid file during cleanup - #14153

Closed
Pytonballoon810 wants to merge 1 commit into
NousResearch:mainfrom
Pytonballoon810:fix/gateway-stale-pid-cleanup-pr
Closed

fix(gateway): remove stale foreign pid file during cleanup#14153
Pytonballoon810 wants to merge 1 commit into
NousResearch:mainfrom
Pytonballoon810:fix/gateway-stale-pid-cleanup-pr

Conversation

@Pytonballoon810

@Pytonballoon810 Pytonballoon810 commented Apr 22, 2026

Copy link
Copy Markdown

Summary

  • Fix stale PID cleanup for ~/.hermes/gateway.pid by unlinking invalid/stale PID files directly.
  • Avoid calling remove_pid_file() in the stale-cleanup path, because that helper intentionally preserves PID files owned by other processes.
  • Add a regression test to ensure stale foreign PID records are removed.

Root cause

get_running_pid() delegates invalid/stale PID cleanup to _cleanup_invalid_pid_path(). For the default PID path, cleanup previously called remove_pid_file(), which refuses to delete files not owned by the current process. A stale foreign PID record could therefore persist and repeatedly block startup with:

PID file race lost to another gateway instance.

Validation

  • python -m pytest tests/gateway/test_status.py -q
  • Result: 28 passed

Files changed

  • gateway/status.py
  • tests/gateway/test_status.py

Impact

Prevents restart/startup flap loops caused by stale foreign PID records after unexpected shutdowns or interrupted replacements.

Stale PID cleanup in get_running_pid() incorrectly called remove_pid_file() for the default gateway PID path. remove_pid_file() intentionally preserves PID files owned by other processes, so stale foreign records could survive indefinitely and cause repeated startup failures with: 'PID file race lost to another gateway instance'.\n\nFix by unlinking invalid/stale PID paths directly in _cleanup_invalid_pid_path().\n\nAdd regression test covering stale foreign PID removal.
Copilot AI review requested due to automatic review settings April 22, 2026 20:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes gateway startup/restart deadlocks caused by stale PID files by ensuring invalid/stale PID records are unlinked directly (rather than delegating to remove_pid_file(), which intentionally preserves PID files owned by other processes during graceful handoffs).

Changes:

  • Update stale/invalid PID cleanup to always unlink() the PID file instead of calling remove_pid_file() for the default PID path.
  • Add a regression test asserting that a stale PID file containing a dead “foreign” PID is removed.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
gateway/status.py Changes stale/invalid PID cleanup to unconditionally unlink the PID file.
tests/gateway/test_status.py Adds regression coverage for removing a stale PID file referencing a dead non-current PID.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread gateway/status.py
Comment on lines +219 to +223
# Invalid/stale PID records must be removed unconditionally.
# Do NOT call remove_pid_file() here: that helper intentionally keeps
# PID files that belong to a different process during graceful
# handoffs, which is the opposite of what stale cleanup needs.
pid_path.unlink(missing_ok=True)

Copilot AI Apr 22, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_running_pid() treats PermissionError from os.kill(pid, 0) the same as ProcessLookupError and calls _cleanup_invalid_pid_path(). With this change, that path now unlinks the PID file unconditionally, which can delete a PID file for a process that is actually still running (EPERM means “process exists but you don’t have permission”). That can allow a second gateway instance to start under the same PID path when the original process is owned by another user / has restricted permissions. Consider handling PermissionError separately (e.g., treat it as “running” and return the PID, or at least skip PID-file deletion on PermissionError and leave the file in place).

Copilot uses AI. Check for mistakes.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels Apr 22, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #13709 — same root cause: _cleanup_invalid_pid_path() delegates to remove_pid_file() which refuses to delete foreign-owned PID files. See also #13947, #14002, #13713.

@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #13709.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the contribution, @Pytonballoon810! This is a well-diagnosed bug report with a correct fix.

However, the identical change has already landed on main — this PR is a duplicate.

  • Commit: 402d048ebfix(gateway): also unlink stale PID + lock files on cleanup
  • File: gateway/status.py_cleanup_invalid_pid_path() now directly calls pid_path.unlink(missing_ok=True) and _get_gateway_lock_path(pid_path).unlink(missing_ok=True), bypassing remove_pid_file() for exactly the reason you identified.
  • Released in: v2026.4.23

As @alt-glitch noted, this is also related to the cluster of issues #13709, #13947, #14002, and #13713 — all fixed by the same upstream change.

This is an automated hermes-sweeper review.

@teknium1 teknium1 closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants