TUI Gateway: pre-compress idle sessions to reduce next-turn latency

[mchen0571]

PR: TUI Gateway: pre-compress idle sessions to reduce next-turn latency

Summary

Adds opt-in idle pre-compression for TUI Gateway sessions. When enabled, the Gateway schedules a quiet background compaction pass after a session has been idle long enough, reducing the chance that the next user prompt blocks on context compression.

Why

Long-running sessions often hit context compression on the next user turn. That puts compression latency on the user's critical path. If the session is already idle after a long response, the Gateway can safely use that idle window to compact context ahead of the next prompt.

Maintainer feedback addressed

Named-profile isolation

Both idle scheduling and the actual worker execution install the live session's profile_home through the HERMES_HOME ContextVar and restore it in finally. No process-global os.environ mutation is used. Regression coverage runs the worker in a real thread against temporary default/named profile homes in both directions.

Busy input remains lossless

idle_compression_running participates in the existing busy-input path. A prompt.submit received during idle compaction is accepted and queued for all queue, steer, and interrupt modes; it is never steered into or used to interrupt the compactor. The already-started compaction completes, its continuation session key is synchronized, and the queued user turn drains exactly once afterward.

Details

• Adds disabled-by-default compression.idle config.
• Derives idle threshold from compression.threshold * 0.9 when no explicit idle threshold is configured.
• Respects parent compression.enabled; disabling global compression also disables idle compression.
• Schedules one debounced idle worker per live TUI Gateway session.
• Uses the live registry object rather than requiring long-lived sid fields on create/deferred session records.
• Uses snapshot/version, normal-turn, finalized, and replacement identity fences before publishing compressed history.
• Serializes session-key synchronization with finalization through a per-session handoff lock: finalization marks its fence first, then waits outside history/registry locks; a blocked lease transfer rechecks liveness and compensates stale lease/notifier/YOLO state before returning.
• Synchronizes session_key after _compress_context rotates the agent continuation session, with atomic live/finalized commit and deterministic Event-barrier race coverage.
• Prevents mutation, eviction, direct steer, and lifecycle paths from racing the same session while idle compaction owns the agent.
• Starts worker threads outside the idle-worker registry lock and performs LLM, sync, and queued-turn drain work outside history/registry locks.
• Uses cooldown and a global semaphore to limit auxiliary compression calls.
• Stops pending workers through current-main's _sid teardown contract.
• Keeps user-facing status quiet by default.

Tests

Latest local validation after rebasing onto NousResearch/main:

python -m pytest -q -p no:cacheprovider -o addopts= \
  tests/test_tui_gateway_idle_compression.py \
  tests/test_tui_gateway_queue_on_busy.py
# 37 passed

python -m pytest -q -p no:cacheprovider -o addopts= \
  tests/test_tui_gateway_server.py
# 386 passed, 1 failed
# The sole failure is test_verification_status_returns_recorded_evidence.
# The identical node also fails on a clean origin/main worktree at
# 3e23c502f2f671582e614d18fb7e6e3ca7fb0260 (expected passed, actual unverified),
# so it is recorded as an existing upstream baseline failure rather than a PR regression.

python -m py_compile \
  tui_gateway/server.py \
  hermes_cli/config.py \
  tests/test_tui_gateway_server.py \
  tests/test_tui_gateway_idle_compression.py

git diff --check origin/main...HEAD
git diff --check
# passed

Scope

This remains an opt-in TUI Gateway feature. It does not change the default compression behavior, production configuration, or non-TUI Gateway services.

[mchen0571]

Rebased this PR onto the latest NousResearch/main and resolved the merge conflicts.

Local verification run after the rebase:

• python3 -m py_compile hermes_cli/config.py tui_gateway/server.py
• python -m pytest tests/test_tui_gateway_server.py -q -o 'addopts=' → 269 passed, 1 warning

Current note: GitHub still shows no check runs/status contexts for this fork PR, so CI appears to be awaiting maintainer workflow approval or not configured for this path.

[teknium1]

Thanks for the careful opt-in design and the snapshot/version guards. The feature is not present on current main, but the current session machinery has moved substantially and exposes two blocking issues.

Problems

• tui_gateway/server.py:2160 / 2227 start the profile-sensitive idle worker without the per-session HERMES_HOME override. Current prompt execution installs that override at tui_gateway/server.py:8935-8937, and resume records profile_home at tui_gateway/server.py:5899-5903. The worker must preserve that profile scope.
• tests/test_tui_gateway_server.py:3074 asserts that a prompt is rejected during compaction. Current prompt.submit deliberately queues busy input through _handle_busy_submit (tui_gateway/server.py:8447-8452); retaining the rejection would regress the no-drop busy-input behavior.

Suggested changes

• Scope the idle worker to session["profile_home"] and add a named-profile integration regression.
• Integrate compaction with the current queued-input path, cancelling/defering compaction safely rather than rejecting the prompt.

Automated hermes-sweeper review.

[teknium1]

This daemon path never installs session["profile_home"] as the HERMES_HOME override. Current prompt execution does so at tui_gateway/server.py:8935-8937; without equivalent scope here, a resumed named-profile session can read the default profile's compression.idle settings and compact outside its profile context.

[mchen0571]

Fixed in 0d23b55. Both idle scheduling and the actual worker install the live session's profile_home through the HERMES_HOME ContextVar and restore it in finally; there is no process-global os.environ mutation. Added real-thread regression coverage for default→named and named→default profile isolation.

[teknium1]

Please do not lock in a rejection here. Current main routes a busy prompt.submit through _handle_busy_submit (tui_gateway/server.py:8447-8452) so input is queued instead of dropped; idle compaction needs to preserve that contract while safely deferring or cancelling the compaction.

[mchen0571]

Fixed in 0d23b55. idle_compression_running now follows the existing busy-submit path: queue, steer, and interrupt modes all accept and queue input without steering into or interrupting the compactor. The started compaction completes, synchronizes the continuation session key, then drains queued input exactly once. Added lossless/busy-mode tests plus finalize/replacement Event-barrier race coverage.

[teknium1]

Thanks @mchen0571 — this was a carefully engineered take on the idle-session compression problem: opt-in design, snapshot/version + replacement-identity fences, per-session handoff lock, real-thread named-profile regression tests in both directions, and the lossless busy-input queueing rework after review.

The trigger half has since landed via #69360: compression.idle_compact_after_seconds compacts long-idle sessions at turn start, routed through the real per-session lock / cooldown / anti-thrash guards, and the motivating issue (#27579) is closed COMPLETED. The compression.idle config namespace here now collides with that key.

The surviving delta — true background pre-compression so the first post-idle turn doesn't pay compaction latency — is a real idea, but this branch predates the per-session compression lock, the deferred engine-notify contract (#69324), lock-skip feedback (#69870), and the quiet-status hooks (#69859), all of which any background compressor must now compose with; the two review blockers (profile-scope gap, busy-input rejection test regressing _handle_busy_submit) also still stand. If resume-time latency proves painful in practice, the right vehicle is a fresh issue/PR against the current contracts rather than keeping this branch alive.

Closing as superseded by #69360, with credit for the design work.