Skip to content

fix(gateway): survive faulthandler.enable() when sys.stderr is None (#71671 salvage) - #72304

Merged
teknium1 merged 3 commits into
mainfrom
hermes/hermes-6ae79939
Jul 27, 2026
Merged

fix(gateway): survive faulthandler.enable() when sys.stderr is None (#71671 salvage)#72304
teknium1 merged 3 commits into
mainfrom
hermes/hermes-6ae79939

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Headless Windows gateways no longer die at startup: faulthandler.enable() on the first line of GatewayRunner.start() raises RuntimeError: sys.stderr is None when the gateway is spawned without a stderr handle (legacy pythonw scheduled tasks, VBS shims, detached stdio='ignore' spawns), taking every platform adapter offline. Root cause: #70344 added the call unconditionally; faulthandler.enable() writes to sys.stderr by default.

Salvages #71671 by @hereicq with authorship preserved. Chosen over the duplicate #72158 (bare skip-guard) because the fallback keeps the #70344 freeze diagnostics working on exactly the headless installs that need them most.

Changes

  • gateway/run.py: wrap faulthandler.enable(); on RuntimeError/ValueError/OSError fall back to enable(file=logs/gateway_faulthandler.log); if even that fails, log and continue (@hereicq)
  • tests/gateway/test_71671_faulthandler_no_stderr.py: behavioral regression test (@hereicq)
  • Follow-up (ours): dropped the PR's second test which regex-asserted gateway/run.py source text — banned antipattern per AGENTS.md "Never read source code in tests"
  • contributors/emails/: mapping for @hereicq

Validation

Check Result
scripts/run_tests.sh tests/gateway/test_71671_faulthandler_no_stderr.py 1/1 passed
E2E: real process with fd 2 closed (sys.stderr is None), isolated HERMES_HOME unguarded enable() raises; guarded path enables via file fallback, dump lands in logs/gateway_faulthandler.log, exit 0
py_compile gateway/run.py ok

Sibling audit: all other faulthandler/stderr sites (gateway/shutdown_watchdog.py, telegram adapter) already wrapped in try/except — run.py:7829 was the only unguarded one.

Closes #71671. Closes #72158 (duplicate, 18h later).

Infographic

PR infographic

hereicq and others added 3 commits July 26, 2026 16:40
faulthandler.enable() writes to sys.stderr by default, and raises
RuntimeError('sys.stderr is None') when the gateway is launched
without an attached console — e.g. via the Windows Startup VBS shim,
pythonw.exe, a detached service, or any parent that redirects stderr
to DEVNULL. Because this happens on the very first line of
GatewayRunner.start(), the whole gateway used to die at startup and
every configured platform adapter (Discord bot, Telegram, Slack, …)
would silently show offline until the user manually re-ran
'hermes gateway run --replace' from a real terminal.

Wrap the call and fall back to a log-file file descriptor
(logs/gateway_faulthandler.log) when stderr is unavailable, so
fatal-error stack dumps still land somewhere useful. If even the
fallback fails we log-and-continue rather than kill the gateway.

Repro traceback (from a real user's gateway-exit-diag.log, launched
via the Startup VBS with stdin_is_tty=false):

    File "gateway/run.py", line 7821, in start
        faulthandler.enable()
    RuntimeError: sys.stderr is None
Source-regex tests are banned (AGENTS.md 'Never read source code in
tests') — keep only the behavioral regression test.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/windows Native Windows-specific behavior or breakage sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Jul 26, 2026
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 37c8e67

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

inline evidence upload failed.

Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants