Skip to content

feat: support fixed token threshold and per-model compression settings - #24704

Open
fredzhuzz wants to merge 1 commit into
NousResearch:mainfrom
fredzhuzz:feat/compression-threshold-tokens
Open

feat: support fixed token threshold and per-model compression settings#24704
fredzhuzz wants to merge 1 commit into
NousResearch:mainfrom
fredzhuzz:feat/compression-threshold-tokens

Conversation

@fredzhuzz

Copy link
Copy Markdown

Description:

Summary

Extends the compression section in config.yaml to support absolute token thresholds and per-model overrides, not just a global percentage.

Changes

New config options (all optional, backward compatible):
- compression.threshold_tokens — absolute token count that overrides the percentage-based threshold globally.
- compression.per_model — per-model overrides map. Each entry can specify threshold (percentage) or threshold_tokens (fixed count).

Priority (highest → lowest):
1. compression.per_model.<model>.threshold_tokens
2. compression.per_model.<model>.threshold
3. compression.threshold_tokens (global fixed)
4. compression.threshold (global percentage, current behavior)

Files modified:
- agent/context_compressor.py — init and update_model() accept optional threshold_tokens parameter
- run_agent.py — reads new config fields, passes to compressor, re-resolves on model switch
- hermes_cli/config.py — DEFAULT_CONFIG + config display update
- hermes_cli/setup.py — interactive setup supports fixed token and per-model configuration
- tests/agent/test_context_compressor.py — 5 new tests covering threshold_tokens logic

Example config.yaml
yaml
compression:
  enabled: true
  threshold: 0.5                         # global default (percentage)
  threshold_tokens: null                 # global fixed token override
  per_model:
    deepseek-chat:
      threshold_tokens: 120000           # deepseek-chat: compress at 120K tokens
    claude-sonnet-4:
      threshold: 0.75                    # claude-sonnet-4: compress at 75%


Closes #24695

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles labels May 13, 2026
@fredzhuzz
fredzhuzz force-pushed the feat/compression-threshold-tokens branch from c3f82ef to 1c2668c Compare May 13, 2026 01:55
Adds three new config options under compression section in config.yaml:

1. compression.threshold_tokens — absolute token count that overrides
   the percentage-based threshold globally.
2. compression.per_model — per-model overrides map, each entry can
   specify threshold (percentage) or threshold_tokens (fixed count).

Closes NousResearch#24695
@fredzhuzz
fredzhuzz force-pushed the feat/compression-threshold-tokens branch from 1c2668c to 01593e5 Compare May 13, 2026 02:01
@fredzhuzz

fredzhuzz commented May 13, 2026

Copy link
Copy Markdown
Author

Closes #24695

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for addressing a real configuration gap. Current main still has only a global user threshold plus built-in model policy (agent/agent_init.py:1502-1549), so the feature remains useful.

Problems

  • The added threshold_tokens= call for selected context engines is incompatible with the current engine contract: ContextEngine.update_model() accepts no such keyword (agent/context_engine.py:215-231). Keep built-in-compressor settings separate from plugin engines.
  • Fixed values must preserve the current effective-input-window invariant. _compute_threshold_tokens() accounts for max_tokens and keeps the trigger below the usable window (agent/context_compressor.py:990-1029); direct assignment can otherwise defer compression past provider overflow.
  • Gateway cached agents would not reload the new settings: construction-time compression keys are explicitly enumerated in gateway/run.py:15615-15634, but this diff does not update that list.
  • The model-switch path re-resolves only fixed tokens, not the proposed per-model percentage override.

Suggested changes

  • Re-target the resolver and precedence logic to agent/agent_init.py:1502-1549, then cover the current runtime transition paths in agent/agent_runtime_helpers.py:1202-1209 and :2017-2024.
  • Add behavior tests for fixed-threshold bounds, plugin engines, model switches, and gateway cache invalidation; update the config example and user docs.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 13, 2026
@teknium1 teknium1 added the area/compression Context compression and continuation sessions label Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants