fix(gateway): persist async task origins across restart - #75117
Conversation
Capture the dispatching turn source, message anchor, profile, and parent session before child construction can mutate context. Persist and restore the snapshot for profile-safe completion routing and compression continuation, while retaining legacy session-key fallback for old rows.
Capture immutable terminal routing before dispatch, checkpoint pending completion state for recovery, and route periodic output directly through the owning profile. Periodic updates remain non-conversational status delivery; only final completion re-enters Hermes as a synthetic turn.
Register producers before exit observers start, persist local notified output through a child-owned sink and exit sidecar, retain undelivered finals during pruning, and acknowledge final delivery only after explicit acceptance.
Assert immutable source/profile routing and inject the thread metadata contract explicitly so the durability PR does not depend on Feishu adapter behavior owned by NousResearch#72024.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tackling a real race: current tools/terminal_tool.py:2552-2561 starts the process before it stamps watcher/notification state at tools/terminal_tool.py:2689-2772, and current routing still falls back to mutable session/cache state in gateway/run.py:20748-20763.
Problems
tools/process_registry.py:1532acknowledges a terminal completion while merely draining/formating it.mark_notification_delivered()deletes its output/exit artifacts and checkpoint state (tools/process_registry.py:1375-1389) before a consumer accepts the synthetic completion. In particular, the TUI can requeue drained events when already busy (tui_gateway/server.py:9655-9660), so the event is retained only in memory after its durable recovery state has been removed.
Suggested changes
- Move terminal acknowledgement out of
drain_notifications()and perform it only after the CLI/TUI successfully accepts the synthetic completion. Retain/requeue the event and artifacts on busy or failure paths, and add coverage for that TUI deferral path.
Automated hermes-sweeper review.
| if text: | ||
| results.append((evt, text)) | ||
| if evt.get("type") == "completion" and _evt_sid: | ||
| self.mark_notification_delivered(_evt_sid) |
There was a problem hiding this comment.
drain_notifications() has only formatted the event here; it has not established consumer acceptance. mark_notification_delivered() deletes the checkpoint and durable output artifacts, but the TUI can still requeue this drained event when its session is busy. Please acknowledge only after the CLI/TUI has successfully accepted the synthetic completion, preserving artifacts on deferral or failure.
Summary
Detached producers currently reconstruct their return route from mutable session/cache state. That breaks after
/new, compression, named-profile routing, or a gateway restart. This PR persists an immutable core origin for background delegation and terminal tasks.The durable origin contains the source snapshot, message anchor, owning profile, session key, and parent physical session. Completion routing validates profile and compression lineage before delivery.
For
terminal(background=true, notify_on_complete=true):This PR is adapter-neutral. Feishu live-thread anchor behavior remains in #72024; Feishu conversation/session identity remains in #75104.
Verification
Known boundaries