Skip to content

fix(gateway): bound hygiene compression and fence late workers - #69866

Merged
teknium1 merged 2 commits into
mainfrom
salvage/53380-hygiene-bounds
Jul 23, 2026
Merged

fix(gateway): bound hygiene compression and fence late workers#69866
teknium1 merged 2 commits into
mainfrom
salvage/53380-hygiene-bounds

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Gateway pre-agent hygiene compression is now bounded: an incoming message never blocks indefinitely on a slow/broken auxiliary compression backend, and a worker that outlives the timeout is fenced so it can never mutate session state late. Root cause: the hygiene pre-sweep awaited run_in_executor with no bound, and wait_for() alone would only bound the await — the executor thread keeps running, so the timeout path needed a commit fence, deferred temp-agent cleanup, and lock-release guarantees.

Changes

  • agent/conversation_compression.py: new CompressionCommitFence (cancellation token with a deterministic commit boundary); compress_context checks the fence immediately before any session mutation — a cancelled worker becomes a no-op that returns the input unchanged, resets _last_compaction_in_place, emits abort telemetry, and releases the per-session compression lock.
  • gateway/run.py: hygiene pre-sweep now uses asyncio.wait_for(asyncio.shield(...), timeout=compression.hygiene_timeout_seconds); on timeout it cancels via the fence (or consumes the result if the worker already crossed the commit boundary), warns the user, records a per-session failure cooldown, and defers temp-agent cleanup until the executor future actually finishes (_defer_agent_cleanup_until_future_done) so live clients are never torn down under a running thread. Per-session failure cooldown also set on compression aborts. Timeout warning delivered via the profile-aware _adapter_for_source.
  • hermes_cli/config.py + website/docs/user-guide/configuration.md: compression.hygiene_timeout_seconds (30) and compression.hygiene_failure_cooldown_seconds (300) knobs.
  • run_agent.py: _compress_context forwards commit_fence.
  • Tests: late-worker-race gateway test (worker completes after timeout → no commit, no leak, cleanup deferred until worker exit), cooldown-skip test, fence unit tests, and a lock-release verification test (fence-cancelled attempt leaves the per-session compression lock free and reacquirable by the next attempt).

Validation

Before After
Slow/broken compression backend inbound handling blocks indefinitely pre-agent bounded at hygiene_timeout_seconds, user warned, turn continues
Worker finishing after timeout could race the live turn and compact SessionDB late fenced — late completion is a no-op, no commit
Temp hygiene AIAgent on timeout leaked (only cleanup path was skipped) cleanup deferred until the worker thread actually exits
Per-session compression lock after timeout at risk of staying held verified released; next /compress acquires and commits

targeted tests: tests/gateway/ -k hygiene 39 passed; tests/agent/test_compression_concurrent_fork.py + tests/gateway/test_session_hygiene.py 71 passed, 0 failed.

Credit

Salvaged from #53380 by @westkite1201; worker fencing added during salvage.

Infographic

hygiene-compression-bounds

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 62fcd86

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

inline evidence is publishing...

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles area/compression Context compression and continuation sessions sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 23, 2026
kshitijk4poor and others added 2 commits July 22, 2026 22:16
… lookup + verify lock reacquire after fence cancel

- gateway/run.py: use _adapter_for_source(source) instead of the raw
  adapters.get(source.platform) map so the compression-timeout warning
  respects transport provenance, relay ingress, and multiplexed profiles
  (matches every other user-facing send in the hygiene block).
- tests: add a lock-release verification regression — a fence-cancelled
  hygiene compression must leave the per-session compression lock free so
  the next attempt (manual /compress retry) acquires it and commits
  normally.
@teknium1
teknium1 force-pushed the salvage/53380-hygiene-bounds branch from a5b3cd3 to 62fcd86 Compare July 23, 2026 05:16
@teknium1
teknium1 merged commit 2e9765b into main Jul 23, 2026
37 checks passed
@teknium1
teknium1 deleted the salvage/53380-hygiene-bounds branch July 23, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants