Skip to content

fix: gracefully handle missing systemctl in container environments - #4415

Closed
dashed wants to merge 1 commit into
NousResearch:mainfrom
dashed:fix/container-systemctl-status
Closed

fix: gracefully handle missing systemctl in container environments#4415
dashed wants to merge 1 commit into
NousResearch:mainfrom
dashed:fix/container-systemctl-status

Conversation

@dashed

@dashed dashed commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Summary

hermes status and hermes gateway status crash with FileNotFoundError when running inside Docker/Kubernetes containers where systemd is not available.

This PR adds container/K8s environment detection and shows proper gateway status using PID-based health checks instead of systemctl.

Before (container)

$ hermes status
...
Traceback (most recent call last):
  File ".../status.py", line 293, in show_status
    result = subprocess.run(["systemctl", "--user", "is-active", ...])
FileNotFoundError: [Errno 2] No such file or directory: 'systemctl'

After (container)

◆ Gateway Service
  Status:       ✓ running
  Manager:      container/K8s

The output matches the systemd path's format exactly (✓ running / ✗ stopped) — the only difference is the manager line showing container/K8s instead of systemd (user).

Problem

On Linux, show_status() and systemd_status() unconditionally run systemctl --user is-active hermes-gateway. The except clause only catches TimeoutExpired, not FileNotFoundError — so containers without systemd get an uncaught exception traceback.

This affects any containerized deployment: Kubernetes pods, Docker containers, LXC, or any environment without systemd as the init system.

Fix

Container detection

New _is_container_env() helper in gateway.py checks:

  • KUBERNETES_SERVICE_HOST env var (injected by K8s into every pod)
  • shutil.which("systemctl") as fallback (catches Docker and other non-K8s containers)

Gateway status in containers

Instead of showing "N/A", the container path uses is_gateway_running() from gateway.status — a PID-based liveness check that works in any environment:

  • Reads gateway.pid from HERMES_HOME
  • Verifies the process is alive via os.kill(pid, 0)
  • Returns True/False → displays ✓ running / ✗ stopped

For systemd_status() (the hermes gateway status command), the container path also shows runtime details from gateway_state.json when available (platform statuses, gateway state).

_is_service_running()

Returns False in container environments — correct because there is no systemd service (the gateway process IS running, but not as a systemd unit).

Changes

File Change
hermes_cli/gateway.py _is_container_env() helper; container-aware systemd_status() with runtime details; _is_service_running() early return
hermes_cli/status.py Container detection + PID-based health check via is_gateway_running()
tests/hermes_cli/test_container_detection.py 10 tests (unit + e2e)

Behavior matrix

Environment Before After
Linux + systemd ✓ running / ✗ stopped No change
macOS + launchd Works normally No change
K8s container Crashes with FileNotFoundError ✓ running / ✗ stopped, Manager: container/K8s
Docker container Crashes with FileNotFoundError ✓ running / ✗ stopped, Manager: container/K8s

Test coverage

10 tests in tests/hermes_cli/test_container_detection.py:

Class Tests What's tested
TestIsContainerEnv 4 K8s env var, missing systemctl, normal Linux, both indicators
TestSystemdStatusContainer 2 Running status in containers, no subprocess calls
TestIsServiceRunningContainer 2 Returns False in containers, calls systemctl otherwise
TestStatusOutputContainer 2 E2E: full show_status() output — container vs normal Linux
uv run pytest tests/hermes_cli/test_container_detection.py -v

Test plan

  • All 10 tests pass
  • hermes status shows ✓ running + container/K8s in K8s pod (verified on live deployment)
  • hermes gateway status shows running state with runtime details in containers
  • Non-container behavior completely unchanged
  • Existing test suite unaffected (no regressions)

@dashed
dashed marked this pull request as ready for review April 1, 2026 08:21
@dashed
dashed marked this pull request as draft April 1, 2026 08:23
@dashed
dashed force-pushed the fix/container-systemctl-status branch from ee7df61 to 68de8a1 Compare April 1, 2026 08:28
@dashed
dashed marked this pull request as ready for review April 1, 2026 08:42
@dashed
dashed force-pushed the fix/container-systemctl-status branch from 68de8a1 to 598a286 Compare April 4, 2026 17:28
hermes status and gateway status crash with FileNotFoundError when
running in K8s containers that don't have systemd/systemctl.

Detect container environments via KUBERNETES_SERVICE_HOST env var
or missing systemctl binary, and show informative N/A output instead
of crashing.

Tests: 10 tests (4 unit for _is_container_env, 2 for systemd_status,
2 for _is_service_running, 2 e2e for full status output).
@dashed
dashed force-pushed the fix/container-systemctl-status branch from 598a286 to 937720b Compare April 7, 2026 11:41
@teknium1

Copy link
Copy Markdown
Contributor

Closing as duplicate — this issue is fixed in #8614 (salvaged from @alt-glitch's comprehensive #8573). Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants