fix(gateway): exit cleanly when another instance detected to avoid launchd restart loop

[liuhao1024]

What does this PR do?

When launchd (macOS) spawns two gateway instances simultaneously (e.g., during display wake or KeepAlive restart), the second instance detects the first via PID file and exits with code 1 (return False → sys.exit(1)). With KeepAlive.SuccessfulExit=false, launchd interprets exit code 1 as a failure and immediately restarts — creating an infinite restart loop that spams logs and drains CPU.

Root Cause

start_gateway() has multiple duplicate-instance guard paths that return False when another instance is detected:

1. First check: get_running_pid() finds an existing instance (line ~15145)
2. Second check: PID file race during startup (line ~15289)
3. Runtime lock check: acquire_gateway_runtime_lock() fails (line ~15297)
4. PID file write race: write_pid_file() raises FileExistsError (line ~15303)

All four paths call return False, which causes sys.exit(1) in the caller. This is correct behavior for actual failures (no platforms connected, etc.), but wrong for "another instance is already running" — that's a clean exit, not a failure.

Related Issue

Fixes #21549

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• See commit messages for detailed changes

How to Test

1. Run pytest tests/ -q — all tests should pass
2. Verify the specific scenario described above is resolved

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 26.4.1

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture and workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A

[gavin-jack]

This PR addresses the macOS launchd double-spawn scenario, but the same root cause
also affects systemd deployments on Linux.

Environment: WSL2 Ubuntu 24.04, hermes-gateway.service with Restart=always,
6 gateway profiles (default + 5 agents), all using gateway run --replace.

Scenario: When systemctl restart hermes-gateway.service is executed, systemd
sends SIGTERM to the old process. If the old process doesn't exit cleanly
(e.g., blocked in asyncio event loop, or drain timeout not reached), the new
process starts while the old one still holds the port. The new process detects
the existing instance via PID file and exits with code 1 — exactly the same
pattern this PR fixes for launchd.

Evidence from journalctl:

hermes-gateway.service: Main process exited, code=exited, status=1/FAILURE
hermes-gateway.service: Scheduled restart job, restart counter is at 75.
❌ Gateway already running (PID 30623).

The restart counter reached 76 within hours.

Suggestion: In addition to the return False → return True fix, consider also
documenting a systemd unit template with an ExecStartPre guard:

[Service]
ExecStartPre=/bin/bash -c 'fuser -k 8649/tcp 2>/dev/null; sleep 1'

This ensures the port is freed before the new process starts, covering cases
where the old process survives SIGTERM but is no longer functional. This is
especially relevant for WSL2 environments where Windows hibernation can leave
stale processes behind.

The code fix in this PR should prevent the exit-1 loop. The ExecStartPre is a
defense-in-depth measure for deployment configurations.

[teknium1]

Closing — the exit-code premise this builds on has been reworked on main since this PR was opened, and the death-spiral it targets has no corroborating reports.

Premise is stale. When this PR was filed (May 7), the concern was "duplicate-instance guard exits 1 → KeepAlive.SuccessfulExit=false → restart loop." Since then the planned-exit/flap path was deliberately reworked:

• fix(gateway): detect launchd in /restart service-manager probe (#43475) #51734 (merged) added the takeover-marker mechanism — the --replace path now writes a marker so the target's shutdown handler recognizes its SIGTERM as a planned takeover and exits 0 instead of 1 (the comment at gateway/run.py explicitly calls out avoiding the Restart=on-failure flap loop).
• [Bug]: gateway launchd KeepAlive (SuccessfulExit:false) doesn't restart after a clean --replace exit on macOS #37388 (closed) and /restart bricks a launchd-managed gateway on macOS — exits 0, KeepAlive.SuccessfulExit=false won't revive it #43475 (closed) tuned the SuccessfulExit/exit-code semantics from both directions.

No corroborating reports. Across the entire launchd/KeepAlive/SuccessfulExit issue history, #21549 is the only report of this specific failure (second instance detects first via PID file → exit 1 → loop), and it was never reliably reproduced ("Hard to reproduce reliably… depends on launchd timing"). The other launchd crash-loops on the tracker are distinct root causes (root-owned lock, missing bootout, EX_CONFIG 78, on-demand-only deferral, etc.).

The real, confirmed double-spawn cause is tracked separately. #42446 (independently reported) → #42450 identifies launchctl kickstart -k racing KeepAlive=true as the actual mechanism that produces two instances. That's being evaluated on its own merits.

Flipping all four duplicate-instance guards to return True also conflicts with #28561, which wants two of those paths (runtime-lock / PID-race) to get stricter liveness checks, not looser exit semantics.

Thanks for the detailed writeup and tests — the analysis was genuinely useful for confirming the area was already addressed. Closing #21549 alongside this.