Skip to content

fix(gateway): persist async task origins across restart - #75117

Open
zhexuany wants to merge 4 commits into
NousResearch:mainfrom
zhexuany:fix/async-origin-durability
Open

fix(gateway): persist async task origins across restart#75117
zhexuany wants to merge 4 commits into
NousResearch:mainfrom
zhexuany:fix/async-origin-durability

Conversation

@zhexuany

Copy link
Copy Markdown

Summary

Detached producers currently reconstruct their return route from mutable session/cache state. That breaks after /new, compression, named-profile routing, or a gateway restart. This PR persists an immutable core origin for background delegation and terminal tasks.

The durable origin contains the source snapshot, message anchor, owning profile, session key, and parent physical session. Completion routing validates profile and compression lineage before delivery.

For terminal(background=true, notify_on_complete=true):

  • periodic output is throttled/coalesced direct status delivery (no synthetic user turn);
  • only final completion re-enters Hermes;
  • local notified processes write output and exit status through a child-owned profile-scoped sink, so a gateway restart can recover new output and the real exit code;
  • undelivered finals remain checkpointed until explicit adapter acceptance or user consumption;
  • fast-exit registration and prune races are covered.

This PR is adapter-neutral. Feishu live-thread anchor behavior remains in #72024; Feishu conversation/session identity remains in #75104.

Verification

  • Core async/terminal focused set: 110 passed
  • Additional CLI/delegation compatibility: 28 passed
  • Ruff, Python compile, and diff checks: passed

Known boundaries

  • Non-host sandbox backends keep their existing reconnect/lost boundary after gateway restart.
  • PTY + durable notification redirects stdout to the child-owned log, which may affect rare CLIs that require stdout TTY detection.
  • Transport-owning relay adapter provenance is not added in this PR.
  • A recovered periodic watcher may repeat one old output tail after restart; final delivery remains deduplicated/durable.

zhexuany added 4 commits July 31, 2026 09:02
Capture the dispatching turn source, message anchor, profile, and parent session before child construction can mutate context.

Persist and restore the snapshot for profile-safe completion routing and compression continuation, while retaining legacy session-key fallback for old rows.
Capture immutable terminal routing before dispatch, checkpoint pending completion state for recovery, and route periodic output directly through the owning profile.

Periodic updates remain non-conversational status delivery; only final completion re-enters Hermes as a synthetic turn.
Register producers before exit observers start, persist local notified output through a child-owned sink and exit sidecar, retain undelivered finals during pruning, and acknowledge final delivery only after explicit acceptance.
Assert immutable source/profile routing and inject the thread metadata contract explicitly so the durability PR does not depend on Feishu adapter behavior owned by NousResearch#72024.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery tool/delegate Subagent delegation tool/terminal Terminal execution and process management sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state needs-decision Awaiting maintainer decision before any implementation labels Jul 31, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling a real race: current tools/terminal_tool.py:2552-2561 starts the process before it stamps watcher/notification state at tools/terminal_tool.py:2689-2772, and current routing still falls back to mutable session/cache state in gateway/run.py:20748-20763.

Problems

  • tools/process_registry.py:1532 acknowledges a terminal completion while merely draining/formating it. mark_notification_delivered() deletes its output/exit artifacts and checkpoint state (tools/process_registry.py:1375-1389) before a consumer accepts the synthetic completion. In particular, the TUI can requeue drained events when already busy (tui_gateway/server.py:9655-9660), so the event is retained only in memory after its durable recovery state has been removed.

Suggested changes

  • Move terminal acknowledgement out of drain_notifications() and perform it only after the CLI/TUI successfully accepts the synthetic completion. Retain/requeue the event and artifacts on busy or failure paths, and add coverage for that TUI deferral path.

Automated hermes-sweeper review.

Comment thread tools/process_registry.py
if text:
results.append((evt, text))
if evt.get("type") == "completion" and _evt_sid:
self.mark_notification_delivered(_evt_sid)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drain_notifications() has only formatted the event here; it has not established consumer acceptance. mark_notification_delivered() deletes the checkpoint and durable output artifacts, but the TUI can still requeue this drained event when its session is busy. Please acknowledge only after the CLI/TUI has successfully accepted the synthetic completion, preserving artifacts on deferral or failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/delegate Subagent delegation tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants