Skip to content

fix(relay): recover corrupted session stacks after failed turn cleanup - #78087

Open
burak33bb wants to merge 1 commit into
NousResearch:mainfrom
burak33bb:agent/fix-relay-session-recovery
Open

fix(relay): recover corrupted session stacks after failed turn cleanup#78087
burak33bb wants to merge 1 commit into
NousResearch:mainfrom
burak33bb:agent/fix-relay-session-recovery

Conversation

@burak33bb

@burak33bb burak33bb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Recover Relay sessions after turn finalization encounters a corrupted LIFO
scope stack.

Interrupted tool calls, provider failures, and incomplete delegated-agent
cleanup can leave an unknown nested scope above the turn scope. Relay then
rejects finalization with:

RuntimeError: scope handle is not at the top of the stack

end_turn() caught and logged this exception, but retained the corrupted
RelaySession in the runtime registry. Every subsequent turn reused the same
broken context, causing repeated finalization failures and sessions that
appeared to hang until the gateway was restarted.

Trying additional scope pops is unsafe because Hermes does not own or know the
handle currently at the top of the native stack. This change instead discards
the affected runtime session and its delegated child sessions. The next turn
creates a clean Relay context while preserving the Hermes conversation and
persisted session state.

This complements #74864, which prevents one known concurrent-turn cause of
scope corruption. This PR handles recovery after corruption has already
occurred, including interruption and provider/delegation cleanup paths.

Related Issue

Fixes #78068.

Also mitigates #78060.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Add RelayRuntime.discard_session_tree() to atomically remove a corrupted
    session and its delegated descendants without attempting further scope pops.
  • Clear discarded sessions' handles and contexts so stale leases fail closed.
  • Invoke recovery when the turn-scope pop fails during end_turn().
  • Add a regression test that injects the native non-LIFO error, verifies parent
    and delegated-child invalidation, and verifies the next acquire creates a
    fresh Relay session.

How to Test

  1. Run the focused Relay suites:

    scripts/run_tests.sh \
      tests/agent/test_relay_llm.py \
      tests/agent/test_relay_tools.py \
      tests/agent/test_auxiliary_relay.py \
      tests/hermes_cli/test_relay_shared_metrics_runtime.py
  2. Confirm the runtime suite reports 15 passed and the command exits with
    status 0.

  3. Run lint on the changed files:

    ruff check agent/relay_runtime.py \
      tests/hermes_cli/test_relay_shared_metrics_runtime.py
  4. Confirm Ruff reports All checks passed!.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Runtime suite: 15 passed
Focused test command exit code: 0
Ruff: All checks passed!
Windows footgun scan: No Windows footguns found (2 files scanned)

@burak33bb burak33bb changed the title fix(relay): recover corrupted session stacks fix(relay): recover corrupted session stacks after failed turn cleanup Aug 4, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Turn finalization crashes with "scope handle is not at the top of the stack" when a tool call is interrupted mid-flight

2 participants