fix(gateway): prevent restart resume fan-out after clean drain

[pvspencer22]

Summary

• persist the one-shot clean-shutdown marker immediately after a successful conversational drain, before slower teardown
• consume that proof immediately after PID/runtime-lock ownership so a failed startup cannot lend the previous process's clean status to a later boot
• preserve exact recovery for sessions interrupted by restart_timeout or shutdown_timeout, including older exact markers whose adapter was unavailable during an intervening clean process
• defer heuristic restart_interrupted recovery until genuine inbound user activity instead of emitting startup turns
• serialize successful-turn recovery clears with shutdown reservation/marking so an older clear cannot delete a newer interruption marker
• clear stale pre-drain recovery state when a chat finishes, including when unrelated cron/API work later times out
• add lifecycle and fan-out regressions for marker consumption, timeout classification, startup failure, mixed exact/heuristic recovery, and clear-versus-mark ordering

Problem

A gateway could drain conversational work successfully and then be terminated during slower teardown before writing .clean_shutdown. The next startup treated the missing marker as evidence of a crash, heuristically marked recently active but idle sessions as restart_interrupted, and injected blank synthetic startup turns across unrelated topics.

Two adjacent lifecycle races mattered as well:

• startup previously consumed the marker only after substantial fallible initialization, allowing a failed startup to leave the predecessor's proof reusable;
• a successful turn's asynchronous recovery-state clear could race shutdown's durable interruption mark and delete the newer marker.

Recovery policy

A known drain timeout identifies actual interrupted work and can justify intentional continuation. A generic crash heuristic cannot prove that a user turn was interrupted, so it preserves the session for the next real inbound message without automatically emitting user-visible startup output.

Clean proof applies only to the immediate predecessor's heuristic crash classification. It does not blanket-suppress older exact timeout evidence. Shutdown reservation and recovery-state persistence are serialized per session so successful completion and interruption marking have a defined order.

The marker remains one-shot. A later clean drain replaces any stale invalidation tombstone, and chats that finish during the drain window have their pre-drain recovery markers cleared even if only cron or API work remains at timeout.

Testing

Fresh verification on the exact refreshed head:

• .venv/bin/python -m py_compile gateway/run.py gateway/session_state.py tests/gateway/test_clean_shutdown_marker.py tests/gateway/test_restart_resume_pending.py
• .venv/bin/ruff check gateway/run.py gateway/session_state.py tests/gateway/test_clean_shutdown_marker.py tests/gateway/test_restart_resume_pending.py
• focused/relevant startup, shutdown, restart, and SessionState cluster: 119 passed
• git diff --check upstream/main...HEAD
• exact cumulative-head independent review: PASS

The focused cluster includes deterministic regressions for startup failure after lock ownership, post-rename marker inspection failure, fail-closed marker invalidation, mixed clean proof plus older exact recovery, failed pre-drain marking followed by successful completion, delayed successful-turn clear versus newer shutdown marking, generic-crash fan-out suppression, drain completion cleanup, and one-shot marker invalidation.

Platform notes

Validated on Linux. Marker timing and startup recovery classification are supervisor-independent and do not introduce platform-specific APIs.

AI Assistance

The implementation, regression tests, and analysis were prepared with AI agents.

Related to #25966

[alt-glitch]

This was generated by AI during triage.

Related: #11099 also moves clean-shutdown marking earlier, but this PR uses a broader exact-marker and clean-drain-consumption policy to distinguish interrupted work from idle-session fan-out. These are related mechanisms, not a duplicate.

[teknium1]

Thanks for the careful lifecycle work. The current-main premise is verified: gateway/run.py:12320-12332 writes .clean_shutdown only after slow teardown, while a missing marker reaches suspend_recently_active() at gateway/run.py:10528-10541; that method marks recent sessions restart_interrupted in gateway/session.py:2693-2701, and current startup scheduling accepts that reason at gateway/run.py:10004-10010.

Problems

• The changed allowlist in gateway/run.py:9789 intentionally stops auto-dispatching restart_interrupted, but website/docs/user-guide/messaging/index.md:724-732 still says those sessions are auto-resumed at startup. The documentation must describe the new exact-marker versus heuristic-marker behavior.

Suggested changes

• Update the restart-resume documentation to say that restart_timeout and shutdown_timeout may receive synthetic recovery, while heuristic restart_interrupted recovery waits for the next real inbound message.

Automated hermes-sweeper review.

[pvspencer22]

Addressed in the latest head (bdb9343). The messaging docs now distinguish exact restart_timeout / shutdown_timeout markers from heuristic restart_interrupted markers, including their different startup recovery behavior.