fix(agent): keep /stop responsive while a large-session compression commit is in flight - #98366
Open
liuhao1024 wants to merge 1 commit into
Open
liuhao1024 wants to merge 1 commit into
liuhao1024 wants to merge 1 commit into
Conversation
…mpression commit During a compression commit the worker retains the commit-fence lock until finish_commit(), and AIAgent.interrupt(hard_cancel=True) admitted the stop via the BLOCKING cancel_before_commit — pinning the stop caller (and the _pending_redirect_lock it holds) behind the whole SessionDB mutation. On a large session (NousResearch#98351) that commit can run for minutes, so /stop and /new stopped answering while the desktop showed a permanent "Summarizing thread" overlay, Ctrl+C's signal handler hung, and the gateway turn-timeout watchdog deadlocked interrupting the very turn it was supposed to reap. Admit the stop through the non-blocking try_cancel_before_commit() instead: when a commit is already in flight the event is published immediately. The commit is still never abandoned mid-mutation, the conversation thread still observes the stop as soon as the compression call returns, and the end state is identical — every stop producer only sets the flag; none consumes "commit settled". The blocking form stays as the fallback for fences without the non-blocking API.
Contributor
Overall: Makes /stop non-blocking during large-session compression commit. What it does
Non-blocking notes
No functional issues found. Non-blocking — please use your judgment. |
Contributor
Author
|
Thanks for the review. On the |
10 tasks
3 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
During a context-compression commit the worker retains the
CompressionCommitFencelock frombegin_commit()untilfinish_commit()— by design, an admitted SessionDB mutation is never abandoned mid-commit. The problem is thatAIAgent.interrupt(hard_cancel=True)admitted the stop through the blockingcancel_before_commit(fence, event), which waits on that same fence lock for the whole commit to finish. On a large session (#98351, 1,500+ messages) the commit (archive/split/rotate) can run for minutes, so every explicit-stop producer was pinned behind it:/stop→tui_gateway._interrupt_session_turn→request_hard_interruptblocked on the fence, and it blocks while holding_pending_redirect_lock, so/new(which also interrupts) stopped answering too — the UI showed a permanent "Summarizing thread" overlay with a dead input box._watch_gateway_turn_inactivity, explicitly designed to stay runnable when gateway asyncio is starved) called the same admission and deadlocked interrupting the very turn it was supposed to reap, so_reap_gateway_turn_processesnever ran.The fence already exposes the non-blocking
try_cancel_before_commit()(plus the lock-freecommit_in_flightobserver) for exactly this reason — gateway session hygiene already uses the non-blocking form; the syncinterrupt()path was the only remaining blocking caller.This PR switches
_admit_hard_cancel()totry_cancel_before_commit():True(cancel won before the boundary) — same as before.False(commit already finished) — same as before.None(commit in flight) — the stop event is published immediately instead of waiting.Semantics are unchanged where it matters: the commit is still never abandoned mid-mutation, the conversation thread still waits for the commit inside the compression call and observes the stop as soon as it returns, and the durable lock is released exactly as before. Every
hard_interruptproducer only sets the flag — none of them consumes "commit settled", so the wait bought nothing while costing UI responsiveness. The blocking form remains as a fallback for fence objects that predate the non-blocking API.Related Issue
Fixes #98351
Type of Change
Changes Made
run_agent.py—_admit_hard_cancel()now admits through the non-blockingtry_cancel_before_commit()and publishes the stop event without waiting behind an in-flight commit; the blockingcancel_before_commitremains as the legacy-fence fallback.tests/agent/test_compression_concurrent_fork.py—test_hard_stop_waits_for_commit_already_admittedupdated totest_hard_stop_returns_promptly_while_commit_in_flight: the stop must return while the commit is still blocked, and the already-admitted commit must still complete fully (compressed transcript lands, stop event set, durable compression lock released) — the end-state assertions from the old test are preserved.How to Test
python -m pytest tests/agent/test_compression_concurrent_fork.py tests/agent/test_compression_review_76354.py tests/agent/test_compression_interrupt_protection.py tests/agent/test_compression_stall_fallback_78981.py tests/run_agent/test_interrupt_propagation.py tests/run_agent/test_steer.py tests/agent/test_interrupt_compat.py -q— 128 passed (Observed result: 128 passed).python -m pytest tests/run_agent/ tests/test_tui_gateway_queue_on_busy.py tests/agent/test_subagent_lifecycle.py -q— 1952 passed; the 7 failures (test_nous_fallback_unavailable,test_run_agentAnthropic shared-client, 3×test_streamingAnthropic callbacks, 2×test_switch_model_reasoning_override) reproduce identically on a cleanupstream/maincheckout — pre-existing, unrelated to this change.archive_and_compactmid-commit, callshard_interruptwhile the commit is in flight, and asserts the stop returns promptly and the commit still completes with the durable lock released. should pass.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass (targeted suites: 128 passed; fulltests/run_agent/+ gateway/interrupt suites: 1952 passed, 7 pre-existing failures reproduced on clean main)Documentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A