fix(gateway): drop --replace + Restart=on-failure in generated service units

[teknium1]

Summary

Hermes-generated gateway service units no longer launch with --replace or revive clean exits — the systemd restart loop in the bug report is gone.

Salvages #23281 and #32989 (both @LeonSGP43) and extends the same fix to the NixOS module.

Changes

• hermes_cli/gateway.py: drop --replace from both systemd unit templates, the launchd plist, and _gateway_run_args_for_profile; flip Restart=always → Restart=on-failure (fix(gateway): stop default replace in service runs #23281 + fix(gateway): avoid systemd restart loops on replace #32989).
• nix/nixosModules.nix: default Restart always → on-failure for the host unit; drop --replace from the container-mode (OCI) container create command.
• Tests: assert --replace absent + Restart=on-failure present across user/system units, launchd plist, and run-args.

Why it's safe

The non---replace start path returns "already running" only when get_running_pid() finds a live gateway — that function checks runtime-lock-active + _pid_exists + start-time match + gateway-process shape, and cleans up stale PID files otherwise (gateway/status.py:1011-1035). After systemd SIGTERMs the old process, the new one sees None and starts clean. The graceful-restart path is untouched: SIGUSR1 → exit 75 → RestartForceExitStatus=75 still respawns. --replace remains available as a manual CLI flag for interactive takeover.

Validation

	Result
tests/hermes_cli/test_gateway_service.py	136 passed
E2E render of real units (systemd user/system, launchd, run-args)	no --replace, Restart=on-failure, exit-75 preserved

Closes #23281, #32989. Original bug report cc'd. Contributor authorship preserved via rebase-merge.

[teknium1]

@benbarclay — pulling you in on the container-mode change since you own the Docker/OCI lifecycle.

Question I wanted to verify before dropping --replace from the container path (nix/nixosModules.nix:977): does the container ever actually need it?

Conclusion: no. Walking the lifecycle:

• preStart runs container create ... hermes gateway run [--replace] only on NEED_CREATE — this bakes the command into the container's PID 1.
• script is container start -a (replays that command); preStop is container stop -t 10 (SIGTERM → 10s → SIGKILL). Outer-service Restart governs respawn.
• The PID file lives on the host-mounted ${stateDir}/.hermes volume, so it persists across container restarts.

The only scenario where --replace could matter is a restart where the previous PID-1 gateway was SIGKILL'd without cleaning up its PID file. But get_running_pid() defuses that natively (gateway/status.py:1011-1035):

1. Runtime lock check first — the lock is an flock held by the old PID 1; the kernel releases it on process exit (even SIGKILL), so is_gateway_runtime_lock_active() returns False → get_running_pid() returns None immediately.
2. Even if a stale record survives — the new container is a fresh PID namespace, gateway is again PID 1. _pid_exists(1) is True (it's init/itself), but the recorded start_time (old container's PID 1) ≠ current_start (new container's PID 1) → the record is rejected → None.

Either path → new gateway starts clean without --replace. And --replace was arguably harmful here: it calls terminate_pid(<recorded_pid>), and inside a PID namespace the recorded pid is 1 — i.e. it'd try to SIGTERM the container's own init.

So removing it is a no-op in the happy path and removes a foot-gun in the SIGKILL-recovery path. Flagging in case you have a container restart edge case I'm not seeing (e.g. a non-host-PID-namespace backend, or a setup where the PID file isn't on the mounted volume). If so, shout and I'll keep it for the container path only.

[github-actions]

🔎 Lint report: hermes/gateway-replace-fix vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9466 on HEAD, 9466 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 4919 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.