Skip to content

fix(gateway): error recovery no longer duplicates accepted inputs (#104653) - #105334

Merged
teknium1 merged 3 commits into
mainfrom
fix/gateway-failure-owner
Sep 7, 2026
Merged

teknium1 merged 3 commits into
mainfrom
fix/gateway-failure-owner

Conversation

@teknium1

@teknium1 teknium1 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Gateway error recovery preserves separately accepted inputs without repeating an input already durably owned by the same delivery or turn.

Refs #104653; complements merged #104899 without claiming to explain historical ID/NULL pairs.

  • Pass an explicit accepted-input marker through the agent's existing persist_user_display_metadata path and the gateway fallback writer. Platform identities include platform/profile/scope/chat/thread plus inbound ID; keyless turns get fresh identities. Raw platform IDs remain unchanged for quote/reply lookup, and marker metadata never reaches provider wire messages.
  • Replace the reviewed ID-only/raw-baseline implementation. A process-local lease does not prove universal ownership: another process's accepted user row cannot suppress this keyless input.
  • Probe existence using narrow SQLite SQL on the published/canonical live compression successor and its ancestors, including compaction archives. Reaped siblings, undone rows, observations, foreign markers, and unmarked rows cannot establish ownership. No whole-history baseline, schema change, or content deduplication.
  • Retain exactly two expanded invariants, the loopback handler/HTTP/SQLite A/B harness, and storage documentation.

Root cause: exception recovery inferred current input ownership from replay text, then from insufficiently scoped IDs/new rows, rather than carrying the current accepted input's identity through persistence.

Validation

Live repro: full _handle_message → production _run_agent/TurnRunner → real AIAgent → loopback HTTP/SSE → on-disk SQLite → persistence/delivery. Same expanded 20-checkpoint fixture: base 869228cab4a8276d3b4c78da9d9939670c47bd0f 7/20, reviewed intermediate 653cd72ef63b013798748774a2509f453b566a77 1/20, fixed 20/20. These are cumulative row-sequence checks, not counts of independent defects. All controlled fault boundaries reached; base/fixed each made 10 loopback requests and zero external connection attempts.

Review blocker/control Reviewed intermediate Fixed
Chat A success ID100, chat B distinct input same ID after resume, constructor failure 1 user row; accepted chat B input lost 2 user rows, raw ID100 preserved on both
Separate process commits nonobserved input during keyless constructor failure Current keyless input lost Both inputs retained
Earlier reaped sibling plus live compression successor Review repro duplicated successor input Live successor and archived root/middle ancestor owners retained, published/map-free routing checked
5,000 archived 4-KiB rows; full keyless failure handler 28,034,300 bytes traced peak, raw scans 107,679 bytes, zero raw scans
Separate identical keyed/keyless inputs and identical timestamps Positive controls retained Every accepted input remains separate
Post-persistence failure / same-delivery pre-agent retry Prior ownership controls No extra row; healthy follow-up preserved
  • Canonical serial runner under the shared lock: 63 passed, 0 failed across 11 files (including silence/retry-replacement controls). Exactly two new invariant tests. Broader directory suites and CI-green are not claimed.
  • Ruff, Windows-footguns, compatibility-pointer, subprocess-stdin, and diff checks passed.
  • The obsolete raw-baseline admission fault was replaced by an ordinary history-read failure control because keyless turns no longer take a baseline.

Limits: constructor and voice-policy faults are controlled injection boundaries, not live messaging-service/vendor failures. Compression checks use real SQLite archives/tree/reroutes, not provider-driven compaction. This is exception-path arbitration, not global exactly-once delivery. Historical rows are unchanged; unmarked historical rows cannot establish ownership on redelivery. No historical data-loss repair or complete explanation of the original report is claimed.

Reproduction commands and fixture boundaries: evals/gateway_failure_ownership/README.md.

Infographic

Gateway failure recovery checks durable ownership while preserving separate accepted inputs

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on b8bda22 — fix(gateway): carry accepted-input ownership through persist

⚠️ Warnings

OSV vulnerability scan · View job

28 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 5m53s vs 5m7s (+15.0%). 5 job(s) slower, 9 faster, 1 unchanged.

  • Python lints / Windows footguns (blocking): +68.0s
  • OS-specific tests / Windows-only tests: -32.0s
  • Docs Site / docs-site-checks: -19.0s
  • OS-specific tests / macOS-only tests: -11.0s
  • Python tests / e2e: -4.0s

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery area/sessions Session lifecycle, resume, persistence, history sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Sep 7, 2026
Namespace delivery markers and assign fresh keyless turn identities instead
of inferring ownership from IDs or process-local row baselines. Query only
marker existence on the canonical live compression continuation and ancestors.
Preserve raw reply IDs and exclude metadata from provider wire messages.

Expand the two existing invariants with resumed cross-chat ID collisions,
a real independent SQLite writer, reaped siblings, and archived-history
allocation controls. All 20 full-handler checkpoints and 63 targeted tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants