fix: drop stale background process session notifications

[Tosko4]

Summary

• expose the active gateway physical session_id through session context as HERMES_SESSION_ID
• stamp background process watcher metadata with the conversation session id at spawn time
• persist/recover that metadata through process-registry checkpoints
• suppress stale background-process watch/completion notifications when a session key has moved to a different physical session, fail closed when stamped session ownership cannot be resolved, and revalidate through final routing and immediately before model/tool or text delivery
• keep notifications alive across normal compression continuations by comparing session ids through the compression-tip chain
• cover stale agent notifications, text-only watcher sends, compression continuations, and checkpoint recovery

Why

Background process notifications can arrive after a user starts a fresh session. Routing by chat/thread alone is not enough: the event may belong to an older physical session while the session key now points at a new one.

Keeping the spawning session_id lets the gateway suppress stale synthetic turns and text-only watcher messages instead of injecting old process output into the new session. Compression continuations are handled separately so a process spawned before auto-compression can still notify the live continuation.

Related Issue

No separate issue. Follow-up slice from closed broad host-support PR #13370; complementary to merged compression-boundary PR #16306.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✅ Tests (adding or improving test coverage)

Changes Made

• gateway/run.py: drops stale or unverifiable stamped process notifications using session_key + captured conversation_session_id, preserves compression continuations, pins the expected process session into synthetic agent events, revalidates after Telegram topic-binding routing overrides and awaited pre-agent setup, and gives stamped process turns a narrow side-effect-free handler path: no inbound enrichment, lifecycle hooks, tool/status/notice/voice/progress/interim callbacks, streaming, generic stable-key postprocessing, goal continuation, or stale response handoff. Stamped notifications wait behind an active foreground turn without steering or interrupting it; while a stamped process agent is executing, ordinary foreground input is queued under both busy-entry paths rather than being steered into or used to interrupt the synthetic turn; dangerous-command approvals auto-deny without a platform prompt in a generation-owned approval namespace; and model-produced MEDIA: directives are intentionally suppressed because this synthetic narrow path is text-only and cannot deliver local files, images, voice, video, or documents. After executor completion, stamped turns return immediately through this narrow result path without heartbeat/inactivity sends, fallback persistence, pending-message dequeue, direct queued-response sends, or recursive foreground follow-up execution. Stamped final delivery runs directly through a per-session boundary lock shared with all session lookups that can auto-reset an expired session, explicit reset/resume/branch switching, and /stop//new generation invalidation, closing the last check-to-adapter.send() race for agent and text-only notifications. Process watcher delivery reconstructs named-profile sources from multiplexed agent:<profile>:... keys and selects that profile's adapter credentials, while retaining default-adapter compatibility for legacy unstamped watchers. It defers model-backed transcript hygiene for synthetic process turns and gates local/proxy model startup on both physical ownership and run generation. Reset invalidates generation before reading the execution slot; process-triggered local/proxy runs expose a concrete interruptible handle plus an identity-owned stamped-process marker before their final execution-boundary check; both identities remain published through backend completion and surrounding synthetic-turn postprocessing, so ordinary foreground input stays behind the existing adapter guard until stamped release and is then drained exactly once instead of adapter hot-looping, steering, or interrupting; explicit /stop and /new can still cancel it, and both identities are generation/identity-CAS-published and identity-cleared together only at the true outer turn boundary. Stamped agent-cache publication is a generation-checked compare-and-set inside the cache lock, stale cleanup identity-evicts only its own cached agent, and stale process runs skip all postprocessing. Old process finalizers generation-guard slot release; one-shot MoA restoration uses exact installed-override ownership so /stop restores the temporary override while /new replacement state wins.
• gateway/slash_commands.py: routes session lookups (including auto-reset), delayed model-picker callback persistence and preflight-warning history reads, explicit reset, resume, and branch session-store mutations through the same process-delivery boundary lock.
• hermes_cli/context_switch_guard.py: makes gateway model-switch warning enrichment async and reads the resolved physical session history inside the runner-owned boundary instead of directly invoking the transition-capable synchronous store.
• gateway/platforms/base.py: exposes a runner-owned async session resolver so adapter middleware can perform transition-capable lookups through the same delivery boundary; secondary-profile adapters receive a profile-bound resolver before middleware runs so lookup and lock keys remain in agent:<profile>. When a foreground adapter task discovers an out-of-band stamped owner, it retains the adapter guard and pending-drain ownership until that stamped lifecycle releases it, then drains the retained foreground event exactly once instead of entering a recursive redispatch loop or exposing an ownerless replay gap.
• gateway/platforms/yuanbao.py: routes recall redaction/patching, observed-group transcript writes, quote-media lookup, and observed-media hydration through that boundary-aware resolver instead of direct synchronous store lookups.
• plugins/platforms/slack/adapter.py: makes Assistant thread session seeding async and boundary-resolved.
• plugins/platforms/telegram/adapter.py: makes unmentioned-group transcript observation async and boundary-resolved across text, location, and media handlers.
• cron/scheduler.py: seeds continuable thread/channel cron sessions through the live adapter's profile-bound resolver on the gateway loop and appends the seed transcript within the same delivery-boundary critical section, preventing cron auto-reset/profile races without adding a second unguarded mirror lookup.
• tools/terminal_tool.py: stamps background process metadata with the spawning conversation session id and flushes the completed notification configuration after spawn so crash recovery does not retain the earlier empty metadata.
• tools/process_registry.py: persists and recovers conversation_session_id for background watcher checkpoints/events and serializes complete checkpoint generations so an older concurrent snapshot cannot overwrite newer metadata.
• tests/gateway/test_background_process_notifications.py: adds stale-boundary, missing-owner/store-failure, compression-continuation, post-validation Telegram topic-rebind, barrier-based reset-during-pre-agent and cache-publication regressions, callback/streaming side-channel suppression checks, direct guarded agent-result delivery coverage, deliberate media-only suppression, proof that stamped turns cannot dequeue or recursively execute queued foreground messages, delivery-first/reset-first boundary-lock schedules, an auto-resetting session-lookup race, a real Yuanbao middleware lookup-versus-delivery schedule, a production busy-session non-interruption regression, a real BasePlatformAdapter regression proving one retained foreground dispatch and one post-release normal turn without recursive task churn, stale deferred-replay suppression, stale local-handle publication CAS preserving replacement ownership, both foreground busy-entry paths retaining input behind a stamped process agent, named-profile adapter selection across injected/completion/running rails, a cron-seed-versus-guarded-delivery lock schedule, a model-switch-warning lookup-versus-delivery lock schedule, queued stamped-event retry coverage, approval auto-denial without visible delivery, and replacement approval-state survival after stale process cleanup.
• tests/gateway/test_proxy_mode.py: proves a stamped process-triggered proxy request exposes a concrete interruptible task handle, publishes the matching stamped ownership marker, queues ordinary foreground input without cancellation in interrupt and steer modes, and cleans both identities after completion or explicit reset cancellation.
• tests/gateway/test_session_state_cleanup.py: proves reset-owned cleanup plus generation-guarded old-turn unwind preserves a newer execution slot, and identity cache eviction cannot remove a newer agent.
• tests/gateway/test_moa_one_shot_restore.py: proves ownership-based restoration removes the still-owned temporary MoA override after /stop, while stale unwind cannot restore old overrides or evict replacement-session cache state after /new.
• tests/gateway/test_slack.py, tests/gateway/test_telegram_group_gating.py, and tests/gateway/test_multiplex_phase0.py: prove adapter middleware awaits the injected resolver rather than the synchronous store and secondary-profile resolution uses the correct profile namespace.
• tests/gateway/test_model_picker_persist.py: proves the delayed picker callback uses the boundary-guarded session lookup before persisting its selected model.
• tests/tools/test_process_registry.py: asserts checkpoint write/recovery preserves conversation_session_id, including a real spawn → stamp → recover lifecycle regression and an ordered concurrent-writer regression.
• tests/cron/test_scheduler.py: covers resolver-based thread/channel seeding, exact reply-key compatibility, in-boundary transcript append, and standalone fallback behavior.

How to Test

1. Start a background process with notify_on_complete=True from a gateway session.
2. Reset or resume that chat/thread so the same session_key points to a different physical session.
3. Let the old process finish: stale output should not be injected into the new session.
4. Repeat with auto-compression instead of reset: output should still notify the live compression continuation.

Validation

• Rebased without conflicts onto origin/main f813c7ddad6f7a4973f83737d36d5e11a5cbbe50; no additional local post-rebase test cycle was run.
• Pre-rebase validation of the exact patch:
  • python3 -m compileall -q gateway/run.py gateway/slash_commands.py gateway/platforms/base.py gateway/platforms/yuanbao.py hermes_cli/context_switch_guard.py cron/scheduler.py plugins/platforms/slack/adapter.py plugins/platforms/telegram/adapter.py tools/process_registry.py tools/terminal_tool.py tests/gateway/test_background_process_notifications.py tests/gateway/test_proxy_mode.py tests/gateway/test_model_picker_persist.py tests/gateway/test_multiplex_phase0.py tests/gateway/test_slack.py tests/gateway/test_telegram_group_gating.py tests/tools/test_process_registry.py tests/cron/test_scheduler.py
  • git diff --check
  • ./scripts/run_tests.sh tests/gateway/test_background_process_notifications.py tests/gateway/test_proxy_mode.py tests/gateway/test_tool_response_drop_recovery.py tests/gateway/test_session_state_cleanup.py tests/gateway/test_moa_one_shot_restore.py tests/gateway/test_agent_cache.py tests/gateway/test_new_clears_last_resolved_model.py tests/gateway/test_session_model_reset.py tests/gateway/test_resume_command.py tests/gateway/test_35809_auto_reset_clean_context.py tests/gateway/test_10710_auto_reset_evicts_cached_agent.py tests/gateway/test_48031_model_switch_after_auto_reset.py tests/gateway/test_35994_reset_button_deadlock.py tests/tools/test_process_registry.py tests/tools/test_terminal_tool.py tests/tools/test_terminal_compound_background.py tests/gateway/test_session_store_runtime_stale_guard.py tests/hermes_state/test_resolve_resume_session_id.py tests/cli/test_moa_command.py tests/gateway/test_usage_command.py tests/gateway/test_title_command.py tests/gateway/test_status_command.py tests/gateway/test_model_command_custom_providers.py tests/gateway/test_undo_rewind_session.py tests/gateway/test_session_info.py tests/gateway/test_session_reset_notify.py tests/gateway/test_compress_command.py tests/gateway/test_model_picker_persist.py tests/gateway/test_model_command_async_offload.py tests/test_yuanbao_pipeline.py tests/test_yuanbao_integration.py tests/gateway/platforms/test_yuanbao_recall_db_only.py tests/gateway/test_yuanbao_media_ssrf.py tests/gateway/test_telegram_group_gating.py tests/gateway/test_slack.py tests/gateway/test_multiplex_phase0.py tests/gateway/test_multiplex_lifecycle.py tests/gateway/test_multiplex_adapter_registry.py tests/cron/test_scheduler.py tests/hermes_cli/test_context_switch_guard.py -q --tb=short — 1,319 passed.
• ./scripts/run_tests.sh -q was attempted against the refreshed branch; the 37k-test run reached 75% before the 10-minute local timeout. Ten failures had appeared outside the touched test files; all touched/focused files above passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched existing PRs/issues for duplicates and superseding fixes
• My PR contains only changes related to this fix
• I've run the full suite to completion — the 37k-test run reached 75% before the 10-minute local timeout; focused validation above passes
• I've added tests for my changes
• I've tested on Ubuntu/Linux

Documentation & Housekeeping

• Documentation update N/A
• cli-config.yaml.example update N/A
• CONTRIBUTING.md / AGENTS.md update N/A
• Cross-platform impact considered
• Tool descriptions/schemas update N/A

Screenshots / Logs

N/A — gateway/process-routing behavior, covered by regression tests.

[Tosko4]

Context for reviewers: this is one of the small follow-up slices from the closed broad host-support PR #13370.

Teknium salvaged the compression-boundary signal into #16306, and the remaining host-side pieces from #13370 are being resubmitted as narrow, independent PRs instead of reviving the original large PR.

This slice is intentionally limited to its described scope; it does not vendor or bundle any external context-engine implementation.

[Tosko4]

Refreshed this against latest main and I still think the bug is relevant.

Why it is still needed:

• current main still routes background-process completions/watch output primarily by session_key / chat-thread routing, so an old process can outlive /new or /resume and report into the newer physical session
• fix: signal compression boundary to context engine #16306 covers the compression-boundary signal; it does not supersede this host-side background process notification boundary
• Improve external context engine plugin support #13370 is still just the old broad closed PR this was split from; this PR remains the narrow background-process slice
• duplicate/supersession searches for conversation_session_id, stale background process notifications, and session-boundary process output did not turn up another merged fix for this exact path

What changed in the refresh:

• rebased onto main at d577408f3f27c99999fafd74ef0cbaae084f7081
• kept the stale-session drop for synthetic agent notifications
• also guarded text-only watcher completion/running-output sends
• allowed normal compression continuations by comparing both session ids through get_compression_tip()
• added checkpoint recovery and compression-continuation regression coverage

Validation:

• python3 -m compileall -q gateway/run.py tools/process_registry.py tools/terminal_tool.py tests/gateway/test_background_process_notifications.py tests/tools/test_process_registry.py
• git diff --check origin/main...HEAD
• ./scripts/run_tests.sh tests/gateway/test_background_process_notifications.py tests/tools/test_process_registry.py tests/tools/test_terminal_tool.py tests/tools/test_terminal_compound_background.py tests/gateway/test_session_store_runtime_stale_guard.py tests/hermes_state/test_resolve_resume_session_id.py -q --tb=short — 221 passed

Full suite note: I also tried ./scripts/run_tests.sh -q; it did not complete cleanly because of unrelated existing failures/timeouts outside this PR’s touched area. I confirmed the Copilot ACP failure reproduces on current origin/main too.

@teknium1 this one should be ready for another look.

[teknium1]

Thanks for the focused session-boundary fix. The underlying issue still exists on current main: _run_process_watcher() routes by session_key (gateway/run.py:15438-15543), while /new replaces the physical ID for that same key (gateway/session.py:2089-2133).

Problems

• Checkpoint persistence is ordered before the new metadata is populated. spawn_local() writes the checkpoint at tools/process_registry.py:745 / 797; the PR assigns conversation_session_id later in tools/terminal_tool.py:2543 (PR right side) without a subsequent write. A crash in that interval recovers an empty ID and the stale guard cannot suppress the event.

Suggested changes

• Flush the checkpoint after all watcher metadata and notification fields are assigned, and add a lifecycle regression that spawns, stamps, recovers, and verifies the captured physical session ID.

Automated hermes-sweeper review.

[Tosko4]

@teknium1 Thanks — your checkpoint-ordering review was right.

While hardening that path, I also found and fixed two related session-boundary races:

• stamped process notifications now fail closed when current session ownership cannot be resolved, and checkpoint generations are serialized so an older snapshot cannot land after newer routing metadata;
• synthetic agent notifications now revalidate the expected physical session at the final gateway handoff, while text-only completion/running updates recheck immediately before delivery. Compression continuations remain valid; /new and /resume boundaries are dropped.

I added deterministic regressions for spawn → stamp → recover, concurrent checkpoint writers, missing/store-failing ownership, and reset-after-initial-validation races.

Current focused validation: 232 passed, plus compileall, Ruff, and git diff --check. I’m running one final exact-head review gate before force-pushing the refreshed commit, and I’ll update this thread with the exact SHA once it lands.

[Tosko4]

@teknium1 Final patch is up. It closes the stale-session notification races across adapter draining, command cancellation, and local/proxy shutdown ownership, with focused regression coverage. Ready for your review.

[teknium1]

Thanks @Tosko4 — the session-boundary kernel of this PR landed in #85939 (rebase-merged): background-process watchers now stamp the spawning session id at spawn and completions are gated through the existing _classify_completion_target policy, so completions from a /new-closed session are dropped. The broader delivery-boundary locks across adapters/cron were deliberately not carried (superseded by the async-delegation boundary machinery that landed since). Credited in the PR body. Closing.