Skip to content

fix(gateway): prioritize user interrupts over internal events - #61858

Open
rungmc357 wants to merge 2 commits into
NousResearch:mainfrom
rungmc357:fix/prioritize-user-over-internal-events
Open

fix(gateway): prioritize user interrupts over internal events#61858
rungmc357 wants to merge 2 commits into
NousResearch:mainfrom
rungmc357:fix/prioritize-user-over-internal-events

Conversation

@rungmc357

Copy link
Copy Markdown
Contributor

Summary

When a gateway session is busy, synthetic internal=True completion events intentionally queue without interrupting the active turn. If one reaches the pending head before a real user follow-up, the user message can still abort the model call but remain behind that internal event in the FIFO. The recursive next turn then handles the synthetic notification first, delaying the user's steering despite the interrupt acknowledgement.

This change gives real user events priority over queued internal events while preserving arrival order within each class. When the bounded queue is full, it prefers dropping a synthetic notification rather than live operator input.

Related follow-ups to #49738 (internal completions do not interrupt busy sessions) and #49776 (busy-session FIFO queueing). I searched existing issues and PRs and did not find an equivalent fix.

Reproduction

  1. Start a gateway turn with busy_input_mode: interrupt.
  2. While it is running, enqueue a background/delegation completion (internal=True).
  3. Send a real user follow-up before the active model call completes.
  4. The model call is interrupted, but plain FIFO dequeues the older internal completion first instead of the user message.

Changes

  • Reorder a newly queued real user event ahead of pending internal events.
  • Preserve user-to-user and internal-to-internal ordering.
  • Preserve existing media merge behavior and the queue depth cap.
  • Add a regression test covering an internal completion already occupying the head slot when a user interrupt arrives.

Test plan

  • scripts/run_tests.sh tests/gateway/test_internal_event_never_interrupts_busy_session.py tests/gateway/test_queue_consumption.py tests/gateway/test_queue_command.py tests/gateway/test_busy_session_ack.py tests/gateway/test_session_race_guard.py
    • 67 passed, 0 failed
  • uv run --extra dev ruff check .
    • passed
  • python3 scripts/check-windows-footguns.py --all
    • passed (753 files scanned)
  • git diff --check origin/main...HEAD
    • passed

Risk and rollback

Risk is limited to ordering in a busy session when real and internal events coexist in the same pending queue. Media-only merging, all-user FIFO ordering, and all-internal FIFO ordering are unchanged. The focused queue tests cover the surrounding interrupt and dequeue paths.

Rollback is a single-commit revert; no configuration, migration, or persistent data changes are involved.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 10, 2026
@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 11, 2026
@rungmc357
rungmc357 force-pushed the fix/prioritize-user-over-internal-events branch from f60bfe6 to a1f3fb1 Compare July 15, 2026 15:52
@rungmc357

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and added focused regression coverage for the busy-session priority invariant.

The new tests exercise the real queue helpers and verify:

  • user input is promoted ahead of queued internal events
  • FIFO is preserved within user and internal event classes
  • a full queue drops or evicts synthetic work before live user steering
  • drain and head-promotion order remain correct
  • photo/media bursts still merge into the user-owned head slot rather than splitting around internal overflow

The media-burst test fails against the prior production commit and passes with the included head-merge fix, so the production delta is required rather than test-only churn.

Verification on current main:

  • 30 focused and sibling tests passed
  • Ruff passed
  • git diff --check passed

rungmc357 and others added 2 commits July 30, 2026 05:52
Queue-level regression coverage for the user-over-internal priority fix:

- Interleaved user + internal arrivals: all user events keep arrival
  order and precede all internal events, pinning the overflow partition
  invariant ([user..., internal...]) the first-internal insertion relies on.
- Drain execution order (dequeue + promote, mirroring the real drain
  site), not just the in-memory slot/overflow arrangement.
- Full _BUSY_QUEUE_MAX_PENDING queue: user steering replaces an internal
  event (head-slot and overflow variants) instead of being dropped, and
  a queue full of user events never sheds one to admit an internal event.
- Handler-level: a user interrupt outranks multiple queued internal
  completions, which cascade after it in arrival order.

The photo-burst test exposed a real regression in the priority fix: with
an internal event in the overflow, a media follow-up was inserted into
the overflow ahead of it instead of album-merging into the user-owned
head slot, splitting one burst into N turns. Fixed by letting the
existing head-slot merge win over the overflow priority insertion — the
head is already ahead of every internal event, so user priority is
preserved either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rungmc357
rungmc357 force-pushed the fix/prioritize-user-over-internal-events branch from a1f3fb1 to dca5bae Compare July 30, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants