Gateway follow-ups queued during a busy/compression window no longer vanish; overflow survives shutdown (#99882, salvage #99912) - #100972
Merged
Conversation
alt-glitch
added
type/bug
…99882) When a follow-up is demoted to /queue during compression-in-flight, it lands in SessionState.conversation.queued_events (overflow) with the slot event in adapter._pending_messages. After the slot's turn completes, _promote_queued_event should move the overflow head into the slot for the recursive drain. When that drain never runs — the #99882 shape: busy window ended through an exit that skipped the promotion site — the overflow is silently orphaned: never dispatched, never persisted, never logged. A 170-char Telegram follow-up vanished without a trace; its re-send also vanished for the same reason. Fix: _rescue_orphaned_overflow stages one orphan into the empty slot on the next idle arrival, and the new message is enqueued behind it so FIFO order (#28503) holds — oldest orphan runs as this turn, the rest drain in order, the new message last. The helper is best-effort (slot occupied or no overflow → no-op) and logs at WARNING when it fires so a future drain regression is visible. Tests (tests/gateway/test_fifo_overflow_rescue.py, 4 cases on the real GatewayRunner FIFO): - moves overflow head to empty slot - no-op when slot occupied - no-op when no overflow - FIFO preserved: orphan-1, orphan-2, new-msg in exact arrival order Existing queue suites pass unchanged (test_queue_consumption — 5 passed). Fixes #99882
Review note on #99912: rescued = 1 followed by if rescued: is a constant conditional — the log block runs unconditionally now that staging is single-orphan by design.
…rder (#99882) Follow-up to the salvaged #99912 rescue. The original helper left the rescued orphan IN the adapter slot while the caller also swapped it in as the current turn, so the post-turn _dequeue_pending_event ran the same follow-up a second time (live repro: TURNS=['Sent','C','C','D']). The helper now pops the oldest orphan and returns it to run as this turn, stages the NEXT orphan in the slot so the drain continues the chain in arrival order, and the call site parks the incoming message behind the chain via _enqueue_fifo (slot when free, overflow otherwise) instead of always appending to overflow. The rescued event's own source drives the turn so reply anchors point at the message actually being answered. Tests: contract updated for the new return type; added the 2-orphan chain case and the single-orphan-then-new-message slot case (both fail against the original helper shape).
…99882) Sibling site of the same loss class. The #72680 shutdown flush only serialised the adapter slot (_pending_messages); the FIFO tail parked in SessionState.conversation.queued_events was discarded with the process, so every follow-up queued behind the head at restart time vanished the same way the idle-orphan did. flush_overflow_to_file writes one payload per overflow event in the slot-flush shape (plus seq for arrival order), so the existing recover_pending_to_db startup replay inserts them with no new reader. Wired into _stop_impl beside the slot flush.
teknium1
force-pushed
the
p1/comp-recovery-fifo
branch
from
September 2, 2026 07:24
24d4f2c to
cffd44d
Compare
Contributor
૮ >ﻌ< ა ci reviewran on cffd44d — fix(gateway): flush the FIFO overflow tail to disk at shutdo
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gateway follow-ups that were queued into the FIFO overflow while a turn was busy (e.g. demoted
interrupt → queueduring in-flight compression) are no longer silently lost when the busy window ends without the post-turn promotion running: the next idle arrival rescues the orphaned chain and runs it first, in arrival order, exactly once — and the overflow tail now survives gateway shutdown alongside the slot.Root cause: the overflow list (
SessionState.conversation.queued_events) is only drained by_promote_queued_eventat the_run_agentpost-turn site. Any exit that skips that site (/stop, generation bump, turn exception, drain) left the tail populated on an idle session, and the fresh_process_messagepath never looked at it.Changes
gateway/run.py—_rescue_orphaned_overflow(new, from fix(gateway): rescue orphaned FIFO overflow when session goes idle (#99882) #99912) + rescue at the idle claim site in_handle_messagebefore_claim_active_session_slot. Follow-up on top of the salvaged commits: the helper now pops the oldest orphan and returns it as the current turn (the original left it in the slot too, so the post-turn_dequeue_pending_eventran it a second time), stages the next orphan in the slot so the drain continues the chain, and the call site parks the incoming event via_enqueue_fifo(slot when free, overflow otherwise). The rescued event's ownsourcedrives the turn.gateway/run.py_stop_impl+gateway/shutdown_flush.py— sibling site of the same loss class: shutdown flushed only the adapter slot ([Bug] Gateway _pending_messages.clear() discards unrecoverable messages on shutdown - FTS corruption data loss #72680);flush_overflow_to_filenow writes each overflow event in the slot-flush payload shape (+seq), so the existingrecover_pending_to_dbreplays them on restart with no new reader.tests/gateway/test_fifo_overflow_rescue.py(from fix(gateway): rescue orphaned FIFO overflow when session goes idle (#99882) #99912, contract updated) — single-orphan removal from both stores, 2-orphan chain staging, occupied-slot / empty no-ops, FIFO across rescue + new arrival, single-orphan-then-new-message lands in slot.tests/gateway/test_shutdown_flush.py— overflow flush ordering/skip cases, round-trip throughrecover_pending_to_db, empty no-op.Validation
/tmp/c1b_repro_fifo.py(realGatewayRunner, realBasePlatformAdapter.handle_message, realSessionStore+SessionDBin temp HERMES_HOME, real compression lock; only the LLM scripted)ORPHAN STATE ... True;FINAL turns=['Sent', 'new idle message D', 'follow-up C (re-sent)']→ C ran only after D's full turn (FIFO violated; any boundary in that window drops it)Rescued orphaned FIFO overflow event ... (#99882)WARNING;FINAL turns=['Sent', 'follow-up C (re-sent)', 'new idle message D'];CLEAN: dispatched exactly once, before DTURNS=['Sent','C','C','D'](double dispatch) → fixed by the follow-up commitpytest tests/gateway/test_fifo_overflow_rescue.py test_shutdown_flush.py test_pending_queue_spool.py test_queue_consumption.py test_queue_command.py test_steer_fifo_overwrite.py test_priority_path_compression_demotion_56391.py test_session_race_guard.py test_restart_drain.py test_goal_continuation_drain.py test_session_stall_watchdog.py test_busy_session_ack.py test_max_concurrent_sessions.pyruff checkon touched filesLive repro: real gateway runner + adapter dispatch, compression lock held during the busy window,
/stopends the window — before: follow-up C orphaned inqueued_eventson an idle session, dispatched only after unrelated message D; after: C rescued on D's arrival, runs exactly once before D, overflow empty.Closes #99882
Salvages #99912 — commits by @salch-cred cherry-picked with authorship preserved.
Infographic