Skip to content

fix(gateway): stop hygiene retry livelock after commit-fence cancel (salvage #96967) - #99514

Merged
teknium1 merged 2 commits into
mainfrom
fix/p1-compression-livelock
Aug 31, 2026
Merged

teknium1 merged 2 commits into
mainfrom
fix/p1-compression-livelock

Conversation

@teknium1

Copy link
Copy Markdown
Collaborator

What does this PR do?

Salvage of #96967 (author: @HexLab98) rebased and composed onto current main, fixing the hygiene-compression retry livelock reported in #96953: after /stop or /restart cancels the compression commit fence, session hygiene treated the cancelled attempt as a clean no-op, recorded no cooldown, and immediately re-armed auto-compression on the next inbound turn — each doomed round waiting toward the 600 s ceiling while inbound messages were demoted interrupt → queue. Real-world impact: gateway unresponsive ~31 minutes with zero compression work committed (#96953, Feishu; independently reproduced on Telegram in the issue thread).

What changed

  • Fence-cancel is detected promptly: the hygiene wait loop re-checks the commit fence on a short 0.25 s poll (composed on top of main's fix(gateway): bound the hygiene-compression turn-hold (salvage of #90845) #92318 turn-hold slice logic, which is preserved unchanged), and hygiene_wait_should_extend() refuses to extend the wait once the fence is cancelled — a cancelled attempt can never ride the 600 s ceiling.
  • Fence-cancel records a cooldown: the timeout unwind captures fence state before try_cancel_before_commit() (which itself sets is_cancelled) so a genuine idle timeout is not mislabeled, and a fence-cancelled attempt persists a hygiene failure cooldown with a distinct reason string (cancelled at commit fence) — the next turn honors the cooldown instead of retrying immediately.
  • Host unwind records a cooldown: a task-cancellation unwind (/restart tearing down the handler) also persists the cooldown across the process boundary (witnessed with a fresh GatewayRunner/SessionDB pair).
  • No sibling hygiene agent while a compression lock is held: hygiene skips spawning a second compressor when one is already in flight.

Composition notes (conflicts vs. current main)

The original PR predates three main-side changes in the same region; all are preserved:

Verification (live before/after)

On main@26f178e5fa, the PR's four behavioral witnesses fail (livelock gap present):

FAILED test_hygiene_fence_cancel_records_cooldown_without_abort_flag
FAILED test_hygiene_does_not_wait_ceiling_after_fence_cancel
FAILED test_hygiene_skips_when_compression_already_in_flight
FAILED test_hygiene_unwind_records_cooldown  — "hygiene unwind did not persist a cooldown; got None"

With this branch: tests/gateway/test_session_hygiene.py + test_hygiene_failure_cooldown_ladder.py + test_compression_in_flight_check.py — 60/60 passed; full tests/gateway/ suite green (the only 2 failures during development were pre-existing buzz-topology fakes, fixed independently on main by e7f6e22c3b; green after rebase).

Fixes #96953
Closes #96967 (salvaged with authorship preserved — both commits are @HexLab98's, cherry-picked/rebased)

Infographic

hygiene compression livelock fix

…96953)

A /stop or /restart abort left hygiene with no cooldown, so the next turn
re-armed auto-compression and waited up to 600s behind a fence that would
refuse the commit again. Record a cooldown on fence-cancel and unwind,
stop extending that wait once the fence is cancelled, and skip a new
hygiene agent while a compression lock is already held.
…#96953)

Prove a fence-cancelled helper (no abort flag) persists cooldown so the
next turn does not re-arm compression, the host does not wait out the
600s ceiling after cancel, a held lock skips the sibling agent, and
unwind cancellation records the same brake.

(cherry picked from commit d2e178c)
@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 806e13d — test(gateway): cover hygiene fence-cancel cooldown and in-fl

⚠️ Warnings

OSV vulnerability scan · View job

6 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 4m9s vs 4m5s (+1.6%). 5 job(s) slower, 6 faster, 1 unchanged.

  • OS-specific tests / macOS-only tests: -56.0s
  • Check contributors / check-attribution: -37.0s
  • Python tests / e2e: -34.0s
  • Python tests / Run tests: -33.0s
  • Python lints / Windows footguns (blocking): +27.0s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Compression commit-fence cancellation + session-hygiene retry livelock — gateway unresponsive ~30 min after /restart on large sessions

2 participants