Skip to content

fix(memory): fail fast on stuck external prefetch - #63002

Closed
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix-61800-external-memory-prefetch-timeout
Closed

LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix-61800-external-memory-prefetch-timeout

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

What does this PR do?

Prevents a wedged external memory provider from stalling every turn during prefetch. MemoryManager.prefetch_all() now runs non-builtin provider prefetch calls behind a short fail-fast timeout and skips the provider while the original stuck call is still hung.

Related Issue

Fixes #61800

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Add an external-memory prefetch timeout knob to agent/memory_manager.py with a 5s default and HERMES_EXTERNAL_MEMORY_PREFETCH_TIMEOUT override.
  • Run external provider prefetch() calls in a daemon thread, returning without blocking when they exceed the timeout.
  • Track still-hung prefetch calls per provider and skip repeated turns until the original stuck call returns.
  • Add a regression test in tests/agent/test_memory_provider.py that simulates a hung external provider and proves the second turn fails fast instead of re-blocking.

How to Test

  1. Run /Users/leongong/Desktop/LeonProjects/worktrees/hermes-agent/.base/.venv/bin/python -m pytest tests/agent/test_memory_provider.py.
  2. Run /Users/leongong/Desktop/LeonProjects/worktrees/hermes-agent/.base/.venv/bin/python -m py_compile agent/memory_manager.py.
  3. Confirm git diff --check is clean.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15.5

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

  • Issue repro shows prefetch() adding a fixed 120s delay per turn when the external HY Memory sidecar hangs.
  • This patch changes that failure mode to a one-time timeout per stuck call, then immediate skips until the hanging call returns.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/memory Memory tool and memory providers P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 12, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Related: #50800 (competing open PR, 2026-06-22) bounds each external provider's prefetch() with an 8s one-shot-executor timeout — same function (prefetch_all) and same core mechanism (fail-fast timeout on external prefetch). This PR is a superset: it adds per-provider stuck-thread tracking to skip repeated turns while a call is still hung. Fixes #61800. A maintainer should pick the canonical approach across #50800/this one.

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for addressing the pre-turn stall. The premise is confirmed on current main: agent/turn_context.py:563 calls MemoryManager.prefetch_all() before the tool loop, and agent/memory_manager.py:507 invokes provider prefetch() synchronously.

Problems

  • The new HERMES_EXTERNAL_MEMORY_PREFETCH_TIMEOUT override is a non-secret behavioral timeout. AGENTS.md:102-106 requires such settings to be configured through config.yaml, not a new user-facing HERMES_* environment variable.
  • The new regression test verifies the repeated-skip state, but after releasing the blocked call it does not issue another prefetch to verify that _stuck_prefetch_threads is cleared and the provider recovers.

Suggested changes

  • Preserve the fail-fast implementation but use the existing config path (or retain a fixed default without a user-facing env override).
  • Add a post-release recovery assertion to the timeout regression test.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 12, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Merged via #62290 (commit 8d1c96f) — your stuck-prefetch fail-fast commit landed on main with authorship preserved via rebase-merge (adapted to a bounded internal guard instead of a HERMES_* env var, per our config policy). Fixes #61800. Thanks @LeonSGP43!

@teknium1 teknium1 closed this Jul 16, 2026
@teknium1 teknium1 added the area/memory Memory subsystem: store, providers, sync, background reviews label Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/memory Memory subsystem: store, providers, sync, background reviews comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: HY Memory server hangs after ~1h — 120s hardcoded prefetch timeout blocks every turn (hermes-hy-memory plugin)

3 participants