[Bug]: Linux default gateway can stay stale after hermes update, causing ImportError

[ccheng555]

Bug description

hermes update updated the shared checkout but silently skipped restarting the running default-profile systemd user gateway. The next Telegram turn loaded a mixture of old in-memory modules and newly updated source files and failed with an ImportError.

Steps to reproduce

1. Run the default profile gateway as the installed Linux systemd user service.
2. Confirm the gateway is active.
3. Run hermes update while the gateway remains active.
4. Send the bot a Telegram message after the update.

Actual behavior

The update started at 2026-08-04T00:02:48, pulled 462 commits, and finished successfully. The update log ends with:

✓ Update complete!

Tip: You can now select a provider and model:
  hermes model

There is no gateway drain or restart step. Earlier updates in the same log include:

→ hermes-gateway: draining (up to 195s)...
✓ Restarted hermes-gateway

The gateway process remained the one started before the update. The next Telegram turn failed with:

ImportError: cannot import name 'is_trivial_prompt' from 'agent.memory_provider'

Trace path:

gateway/run.py -> _run_agent -> _run_agent_inner
run_agent.py: from agent.memory_provider import is_trivial_prompt

A manual hermes gateway restart immediately fixed Telegram.

Expected behavior

As documented, hermes update should detect and restart every running gateway that uses the updated checkout. If restart detection or restart itself fails, the update should exit nonzero or print a clear warning and manual recovery command. It should not report a clean update while leaving a stale gateway running against replaced source files.

Environment

• Hermes Agent: v0.20.0 (2026.8.3) after update
• Host: WSL2 Linux
• Gateway: default profile, installed systemd user service
• Python: 3.11.15
• SQLite: 3.53.1
• Gateway persistence: systemd linger enabled

Additional evidence

The same update.log demonstrates that gateway auto-restart worked on earlier updates but was omitted entirely from this run. This is the default profile on Linux, so it is distinct from reports limited to stale non-default profiles or macOS launchd behavior.

[monerostar]

Code-path trace on Ubuntu 26.04 linux-5800x (kernel 7.0.0-28-generic). Did not re-run the full WSL2 repro here.

Likely gate: supports_systemd_services() (hermes_cli/gateway.py). On WSL it returns _wsl_systemd_operational(), which only calls _systemd_operational(system=True) i.e. systemctl is-system-running on the system bus. Accepted states are running/degraded/starting/initializing. Anything else (timeout, offline, unexpected string) makes supports_systemd_services() false.

When that is false, hermes update skips the whole systemd unit restart branch in hermes_cli/update_cmd.py (the list-units hermes-gateway* + drain/restart path). That matches a clean Update complete with no draining/Restarted lines even though a user-unit gateway was still up.

Fallback is the manual PID path. That can still miss a clean profile-mapped restart if discovery and profile PID-file mapping disagree (find_profile_gateway_processes reads profile.path/gateway.pid; write_pid_file uses HERMES_HOME/gateway.pid). On profile-scoped HERMES_HOME installs those often match; on mixed layouts they may not. Either way, a silent skip of the systemd branch is enough to leave a stale gateway process against newly updated source and produce ImportError on the next turn.

Suggested fix direction:

1. Make _wsl_systemd_operational() true if either system or user systemd is operational (mirror the container helper that already checks both).
2. When systemd discovery fails on WSL, do not fail open into silence: warn that a running gateway may be stale and print hermes gateway restart.
3. Add a test for WSL + user-systemd-only (current tests/hermes_cli/test_gateway_wsl.py mostly mocks the WSL systemd helper).

Happy to take a small PR for (1)+(2) if maintainers want that scope.

[monerostar]

Follow-up: open PR #78590 addresses the silent whole-phase restart failure (debug-only except) with warn + nonzero exit. Linux verify on 5800x: pytest tests/hermes_cli/test_update_gateway_restart_aborted.py 5 passed. Prefer that over a competing monerostar PR for this issue.

Residual WSL supports_systemd_services() false-negative (no exception) can still be a later complementary patch if needed after #78590 lands.

[antti-jpg]

Additional repro: standard Ubuntu (not WSL), default + sibling profile both remain stale; wrapper emits false “Gateway restarted”

This reproduces on a regular Ubuntu systemd host and affected two running systemd user gateway units, not just WSL/default-profile detection.

Environment

• Ubuntu Linux, kernel 6.8
• Hermes git install at /usr/local/lib/hermes-agent
• systemd user gateways with linger enabled
• default gateway plus one sibling profile gateway
• update from Hermes 0.19.1 to 0.20.0, 1,224 commits

Timeline and evidence

Before the update:

hermes-gateway         PID 991946, started 2026-08-07 07:07:35 UTC
hermes-gateway-fitness PID 934830, started 2026-08-03 03:33:45 UTC

The system-level auto-update service ran:

2026-08-10 03:34:52 UTC start
2026-08-10 03:37:43 UTC exit 0

The update log confirms the checkout and package moved to 0.20.0, then ends with:

✓ Update complete!
✓ Installed hermes-acp launcher → /usr/local/bin/hermes-acp

There are no draining or Restarted hermes-gateway* lines for this update run.

After the update, both units still had the exact same PIDs and pre-update ExecMainStartTimestamp values shown above. They were running old in-memory code against the newly replaced checkout.

The wrapper then sent the user a hardcoded success message saying Gateway restarted because hermes update exited 0. That statement was false.

Impact

• Both profile gateways continued on stale code after a successful update.
• The user received a false all-clear.
• Cross-version lazy imports can create mixed-runtime failures.
• Cron and delivery debugging becomes unreliable because the on-disk source/version no longer identifies the running gateway code.

Suggested postcondition

After an update that changes source/runtime:

1. Enumerate every running gateway tied to the checkout.
2. Restart each one.
3. Verify each PID or ExecMainStartTimestamp changed and the new process reports the new version.
4. If any expected gateway did not restart, exit nonzero and print the exact manual recovery command.
5. Do not let wrappers infer “restarted” from update exit code alone; expose a machine-readable restart result.

This is the same failure family as #78574, but confirms it on standard Ubuntu and across default + sibling profiles.