Skip to content

fix(compression): let explicit interrupts cancel safely - #74449

Closed
suparious wants to merge 1 commit into
NousResearch:mainfrom
suparious:fix/explicit-compression-interrupt
Closed

fix(compression): let explicit interrupts cancel safely#74449
suparious wants to merge 1 commit into
NousResearch:mainfrom
suparious:fix/explicit-compression-interrupt

Conversation

@suparious

Copy link
Copy Markdown
Contributor

What does this PR do?

Makes interrupt-protected context compression cancellable by an explicit user or lifecycle stop, without weakening protection against ordinary incoming messages, voice interjections, or active-turn redirects.

The reproduced CLI failure was:

  1. context compression held the session compression lease while generating its handoff summary;
  2. the user pressed Ctrl+C;
  3. interrupt protection kept the summary worker alive after the frontend's grace period;
  4. the CLI accepted another turn while that worker still held the lease;
  5. the next persistence operation failed closed with session storage could not be written.

This change:

  • separates explicit hard cancellation from ordinary interrupt/redirect state with a dedicated threading.Event;
  • introduces AuxiliaryExplicitCancellation, an attempt-local frozen-cause signal that cannot be confused with provider/plugin/OS InterruptedError;
  • isolates only the synchronous provider callback and stream aggregation in a bounded daemon worker during protected compression, allowing the owning transaction to unwind promptly without closing or evicting process-shared OpenAI-compatible, Codex, Anthropic, or Bedrock clients;
  • propagates caller ContextVar state plus provider progress and interrupt-protection hooks into the isolated call;
  • atomically linearizes Codex timeout cleanup against explicit cancellation, preventing a mixed outcome where the owner reports cancellation after the shared-client timeout path has already won;
  • propagates hard cancellation through child agents and explicit stop surfaces: CLI Ctrl+C/signals, gateway/API/ACP stop and shutdown, SSE disconnect/cancellation, inactivity timeouts, cron, lifecycle cancellation, and foreground/background delegation cancellation;
  • keeps old/synthetic/third-party agent compatibility through statically feature-detected hard_interrupt() with a legacy interrupt(message=None) fallback;
  • serializes hard-cancel admission against compression commit admission with CompressionCommitFence, so cancellation either wins before mutation or waits for an already-admitted commit to finish;
  • aborts before session rotation or late DB commit, restores in-place transcript mutations and an explicit snapshot of compressor summary/provenance/count/savings/cooldown/error/fallback/progress/telemetry state, and releases the heartbeat and compression lease;
  • snapshots and verifies the exact raw durable cooldown row under the session lease (including expired/null states), and surfaces compensating-write failures instead of reporting a false no-op;
  • preserves the established failure/fallback behavior for unrelated InterruptedError instances;
  • adds deterministic cross-thread regressions for provider-call isolation, shared-client safety, frozen cancellation cause, hard-versus-soft interruption, compatibility fallback, pre-commit races, full rollback, lock release, recovered append, and successful retry.

A cancelled provider daemon can outlive its owner until the request's existing provider timeout. It cannot access transcript/commit state or retain the session lease, and no shared client is closed or evicted to stop it.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

Post-rebase verification against current NousResearch/hermes-agent:main:

  • canonical affected suite: 1,178 passed, 0 failed across 63 isolated test files after rebasing onto the upstream test-pruning sweep;
  • compression concurrency/rollback file: 28 passed, 0 failed;
  • TUI protocol/compatibility file: 39 passed, 0 failed;
  • Ruff: passed for all 25 changed files;
  • Windows footgun scan: passed for all 25 changed files;
  • git diff --check: passed;
  • added-line secret/injection/debug scan: passed.

A repository-wide run also exercised over 50,000 tests. It exposed eight patch-related compatibility failures; all were corrected and are covered by the green affected suites above. Remaining repository-wide failures were unrelated profile opt-out/cache test-environment failures in unchanged skill/Honcho paths.

Test Configuration

  • OS: Linux
  • Python: 3.12.13 via uv
  • Test runner: scripts/run_tests.sh (per-file isolated subprocesses)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective
  • New and existing tests covering the changed behavior pass locally
  • Any dependent changes have been merged and published in downstream modules

Documentation and downstream dependency changes are not applicable: this is internal interrupt, provider-call isolation, and compression-transaction behavior.

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/acp Agent Communication Protocol adapter comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/tui Terminal UI (ui-tui/ + tui_gateway/) tool/delegate Subagent delegation area/compression Context compression and continuation sessions sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state needs-decision Awaiting maintainer decision before any implementation labels Jul 29, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to merged #49928: that patch masks ordinary incoming-message interrupts during compression; this patch adds a distinct fence-linearized explicit-stop cancellation path across frontends.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for separating explicit cancellation from ordinary incoming-message interruption. The underlying gap remains on current main: CLI Ctrl+C uses interrupt() at cli.py:15457-15466, compression enters aux_interrupt_protection() at agent/context_compressor.py:3591-3599, and the Codex stream suppresses interruption while that protection is active at agent/auxiliary_client.py:1183-1188.

No concrete blocking implementation defect was confirmed by static review of the PR diff.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 30, 2026
teknium1 added a commit that referenced this pull request Aug 1, 2026
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR #76354 review, blocking finding 2 / merge gate 2.
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Aug 2, 2026
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.

Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.

Based on NousResearch#74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Salvaged via #76647 with your commit cherry-picked and authorship preserved — thank you! Your fix is exactly right: the hard-cancel Event, the daemon-isolated protected provider call, and the full cooldown/lease rollback all survived review intact (174 tests, mutation-checked, E2E-verified).

The salvage rebases onto current main (your branch had drifted 51 commits and picked up some conflict-resolution damage in tui_gateway/methods_session.py), and adds one follow-up commit widening the same hard-stop semantics to _abandon_timed_out_gateway_turn, a timeout surface that landed on main after your branch point.

#76647 is armed to merge on green CI. Closing this one in its favor.

kshitijk4poor pushed a commit that referenced this pull request Aug 2, 2026
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.

Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.

Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
teknium1 added a commit that referenced this pull request Aug 2, 2026
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR #76354 review, blocking finding 2 / merge gate 2.
teknium1 added a commit that referenced this pull request Aug 2, 2026
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR #76354 review, blocking finding 2 / merge gate 2.
@suparious
suparious deleted the fix/explicit-compression-interrupt branch August 5, 2026 00:09
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.

Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.

Based on NousResearch#74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR NousResearch#74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; NousResearch#74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR NousResearch#76354 review, blocking finding 2 / merge gate 2.
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR NousResearch#74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; NousResearch#74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR NousResearch#76354 review, blocking finding 2 / merge gate 2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions comp/acp Agent Communication Protocol adapter comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/tui Terminal UI (ui-tui/ + tui_gateway/) needs-decision Awaiting maintainer decision before any implementation P1 High — major feature broken, no workaround sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/delegate Subagent delegation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants