Skip to content

Fix systemd timeout check for missing user unit - #57901

Closed
rodion-m wants to merge 1 commit into
NousResearch:mainfrom
rodion-m:fix/systemd-timeout-loadstate
Closed

Fix systemd timeout check for missing user unit#57901
rodion-m wants to merge 1 commit into
NousResearch:mainfrom
rodion-m:fix/systemd-timeout-loadstate

Conversation

@rodion-m

@rodion-m rodion-m commented Jul 3, 2026

Copy link
Copy Markdown

Summary

  • query LoadState together with TimeoutStopUSec in check_systemd_timing_alignment()
  • skip user-scope systemd results whose unit is not-found, masked, or error before falling back to the system unit
  • add regression coverage for a system service where systemctl --user show hermes-gateway returns LoadState=not-found plus the user-manager default timeout

Why

On a system-level Hermes gateway deployment, the current check tries systemctl --user show hermes-gateway --property=TimeoutStopUSec first. On Ubuntu/systemd, that can return success for a missing user-scope unit with LoadState=not-found and the manager default TimeoutStopUSec=1min 30s. Hermes then logs a false stale-unit warning even when the real system service has an aligned timeout.

Observed on a live v0.18.0 system-service gateway:

  • user-scope query: LoadState=not-found, TimeoutStopUSec=1min 30s
  • system query: effective timeout >= drain timeout + headroom
  • warning before this patch: Stale systemd unit detected: hermes-gateway.service has TimeoutStopSec=90s...

Fixes #36755.

Test

  • pytest tests/gateway/test_shutdown_forensics.py -q

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have duplicate This issue or pull request already exists labels Jul 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #34734 — same mechanism (query LoadState alongside TimeoutStopUSec in check_systemd_timing_alignment() and skip user-scope units whose LoadState is not-found/masked/error). #34734 is the earliest open PR using this approach; #36766 and #37324 are the same fix. #54396 solves the same #36755 false-positive via a different mechanism (detect the systemd manager from /proc/self/cgroup) and is related rather than a duplicate. Flagging the cluster so a maintainer can pick one.

@rodion-m

rodion-m commented Jul 3, 2026

Copy link
Copy Markdown
Author

Closing as duplicate after triage. This fix belongs to the existing upstream cluster for #36755; #34734 is the earliest open PR using the same LoadState-based approach, and #37324 carries the same approach with regression coverage. Our production verification still confirms the bug is present in v0.18.0, but keeping another duplicate PR open would add noise for maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(diagnostics): check_systemd_timing_alignment false-positive — systemctl show returns default 90s for nonexistent --user unit

2 participants