Skip to content

fix(gateway): detect stale sys.modules via expected exports (#17648 follow-up) - #19531

Closed
anuragbanerjee wants to merge 1 commit into
NousResearch:mainfrom
anuragbanerjee:fix/gateway-stale-sys-modules-exports
Closed

fix(gateway): detect stale sys.modules via expected exports (#17648 follow-up)#19531
anuragbanerjee wants to merge 1 commit into
NousResearch:mainfrom
anuragbanerjee:fix/gateway-stale-sys-modules-exports

Conversation

@anuragbanerjee

@anuragbanerjee anuragbanerjee commented May 4, 2026

Copy link
Copy Markdown

What does this PR do?

After hermes update, a gateway can keep running with old modules still in sys.modules while tools on disk import newer symbols. Users see import errors; session commands do not reload Python.

#18409 restarts the gateway when key source files look newer than at startup (mtime check). This change adds a second condition: if hermes_cli.config or the project utils package is already loaded but is missing cfg_get, atomic_replace, or base_url_host_matches, treat the process as stale and use the same restart path. It does not add broad try/except import fallbacks (#17935).

Related Issue

Refs #17648. Follow-up to #18409.

Source branch

Fork: github.com/anuragbanerjee/hermes-agent, branch fix/gateway-stale-sys-modules-exports. To check out locally: gh repo clone anuragbanerjee/hermes-agent -- -b fix/gateway-stale-sys-modules-exports (or add the fork as a remote and fetch that branch).

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • gateway/run.py: _loaded_modules_missing_expected_exports(); call from _detect_stale_code(); warn when boot mtime baseline is zero.
  • tests/gateway/test_stale_code_self_check.py: tests for the export-based path.

How to Test

  1. pytest tests/gateway/test_stale_code_self_check.py -v
  2. Optional: long-lived gateway + update without process exit; next message should restart or succeed, not fail importing cfg_get.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform:

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

N/A

Screenshots / Logs

N/A

@anuragbanerjee
anuragbanerjee force-pushed the fix/gateway-stale-sys-modules-exports branch 2 times, most recently from 0c22930 to f080954 Compare May 4, 2026 05:07
…usResearch#17648)

- Add _loaded_modules_missing_expected_exports() for cfg_get / utils helpers
- Run export check before mtime comparison so live-update mismatches restart
- Log when mtime baseline is zero but boot time was captured
- Extend stale-code self-check tests
@anuragbanerjee
anuragbanerjee force-pushed the fix/gateway-stale-sys-modules-exports branch from f080954 to 26837b5 Compare May 4, 2026 05:10
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels May 4, 2026
@anuragbanerjee
anuragbanerjee marked this pull request as ready for review May 4, 2026 07:14

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the stale-module failure and adding focused coverage.

Problems

  • Current main deliberately removed this exact per-message stale-code auto-restart mechanism in f2332d4f1 because it interrupted chats and dropped the triggering message. The current design instead records a boot fingerprint in gateway/run.py:20274-20279 and limits the guard to model switching in gateway/slash_commands.py:57-83.
  • The proposed export list is incomplete: tests/test_stale_utils_module_import.py:52-63 reproduces the same failure when stale utils lacks env_float, which is not checked by the new helper. The linked June 24 report also identifies stale imports from tools.tool_backend_helpers and cron.jobs.

Suggested changes

  • Re-scope any follow-up to current gateway/code_skew.py behavior without restoring automatic restarts that drop arbitrary inbound messages.
  • Cover representative lazy-import consumers rather than treating three exports as a complete stale-module contract.

Automated hermes-sweeper review.

Comment thread gateway/run.py
if utils_mod is not None:
if not hasattr(utils_mod, "atomic_replace"):
return True
if not hasattr(utils_mod, "base_url_host_matches"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixed list is not a complete stale-module contract: current regression coverage at tests/test_stale_utils_module_import.py:52-63 reproduces the same ImportError when stale utils lacks env_float, which this probe does not check. The linked June 24 report also identifies missing exports outside utils.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 12, 2026
@anuragbanerjee

Copy link
Copy Markdown
Author

Closing as superseded — the stale-code detection approach on main moved to git-revision fingerprinting (gateway/code_skew.py) instead of export-based sys.modules checks, and the auto-restart path this PR targeted was removed. The underlying gap (stale-module risk on regular agent turns, not just model switching) is worth tracking separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants