Skip to content

fix(compression): respect user-configured threshold below 0.50 on small-ctx models - #66249

Open
AlexFucuson9 wants to merge 2 commits into
NousResearch:mainfrom
AlexFucuson9:fix/compression-threshold-user-config
Open

fix(compression): respect user-configured threshold below 0.50 on small-ctx models#66249
AlexFucuson9 wants to merge 2 commits into
NousResearch:mainfrom
AlexFucuson9:fix/compression-threshold-user-config

Conversation

@AlexFucuson9

@AlexFucuson9 AlexFucuson9 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

The small-context threshold floor (_SMALL_CTX_THRESHOLD_PERCENT = 0.75) forces any model under 512K context to compress at ≥75% of the window. This overrides explicit user configuration like threshold: 0.3 (30%).

For a 160K-ctx model with 100K TPM rate limits:

  • User sets threshold: 0.3 → expects compression at 48K tokens
  • System forces 75% → compression triggers at 120K tokens
  • Session stays under 120K → zero compression events across 22 API calls
  • At 55K+ context, every API call exceeds the 100K TPM budget → cascading 429s

Fix

Only apply the 75% floor when the configured threshold is at or above the default (0.50). A value below 0.50 signals deliberate user configuration — respect it.

The floor still protects default-config users (0.50 → raised to 75% on small-ctx models) while allowing power users to compress earlier for rate-limit compliance.

1 file, 12 lines added, 1 line changed.

Closes #66177

@AlexFucuson9
fix(codex): hash over-length prompt_cache_key to fit 64-char limit
defdfa2 
The Codex/OpenAI backend rejects prompt_cache_key > 64 chars with
HTTP 400 BadRequestError. When _content_cache_key falls back to
session_id (no static instructions+tools to hash), the key can be
arbitrarily long (e.g. paperclip:company:<uuid>:agent:<uuid> at 97
chars). The 400 is masked by the fallback chain, making the primary
provider look unused.

Fix: after computing cache_key, hash it to pck_<sha256[:24]> when
it exceeds 64 chars. hashlib is already imported. The xAI extra_body
path also benefits since it reads the same cache_key variable.

Closes NousResearch#66045
@AlexFucuson9
fix(compression): respect user-configured threshold below 0.50 on sma…
4fbf72d 
…ll-ctx models

The small-context threshold floor (_SMALL_CTX_THRESHOLD_PERCENT = 0.75)
raises any model under 512K context to at least 75% threshold. This is
a safety net to prevent thrashing on 128K-262K models at the default
50% trigger.

But when a user explicitly sets threshold: 0.3 (30%) to stay within
TPM rate limits (e.g. 100K TPM on a 160K-ctx reasoning model), the
floor overrides their config to 75% (120K tokens). Compression never
fires, context grows to 55K+, and every API call exceeds the 100K
TPM budget — cascading 429 errors.

Fix: only apply the 75% floor when the configured threshold is at or
above the default (0.50). A value below 0.50 signals deliberate user
configuration — respect it.

Closes NousResearch#66177
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles area/billing Account usage, credit usage, billing (cross-cutting) provider/openai OpenAI / Codex Responses API provider/xai xAI (Grok) codex needs-decision Awaiting maintainer decision before any implementation sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) labels Jul 17, 2026
@alt-glitch

alt-glitch commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #66177 and the Codex cache-key fixes #24273/#66245. This PR bundles two independent changes (small-context compression semantics and over-length cache-key hashing); please consider splitting them for review.

tonydwb
tonydwb reviewed Jul 17, 2026
View reviewed changes

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved (comment only — token read-only)

Clean, well-scoped fix. The logic change is correct — applying the compression threshold floor only when the configured value looks like the default (>= 0.50) preserves intentional user configuration. Documentation clearly explains the rationale.

Reviewed by Hermes Agent

tonydwb
tonydwb reviewed Jul 17, 2026
View reviewed changes

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved (LGTM)

Overview

Respects user-configured threshold below 0.50 on small-ctx models for compression. +18/0.

Security

  • No hardcoded secrets or credentials

Code Quality

  • Clean threshold handling

Looks Good

  • Minimal feature fix

Reviewed by Hermes Agent

@Li20030706 Li20030706 mentioned this pull request Jul 17, 2026
Open
@teknium1

teknium1 commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Thanks for identifying the configured-threshold case. The compression premise is still present on current main: agent/context_compressor.py:1221-1222 unconditionally raises every sub-512K configured threshold to 75%, including 0.30.

Problems

  • The bundled Codex hunk is already superseded. Current main's 61bbc3933091dc418d683aed2928063e88be694e centralizes bounding in agent/transports/codex.py:16-27 and applies it after middleware preflight as well as at build time; its regression coverage includes top-level and xAI boundary paths in tests/agent/transports/test_codex_transport.py:380-411.
  • The compression change needs a regression test. Existing floor coverage in tests/agent/test_compression_small_ctx_threshold_floor.py:29-44 covers 0.50 and 0.85 but not a below-default configured threshold.

Suggested changes

  • Salvage the compression hunk independently and omit the already-implemented Codex commit.
  • Add a sub-512K threshold_percent=0.30 test asserting the effective percentage and derived threshold remain at 30%.

Automated hermes-sweeper review.

@teknium1 teknium1 added the sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit label Jul 18, 2026
@alt-glitch alt-glitch mentioned this pull request Jul 19, 2026
Closed
@Sora-bluesky Sora-bluesky mentioned this pull request Jul 19, 2026
Closed
13 tasks
@teknium1 teknium1 added the area/compression Context compression and continuation sessions label Jul 19, 2026
@GottZ GottZ mentioned this pull request Jul 27, 2026
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@tonydwb tonydwb tonydwb left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/billing Account usage, credit usage, billing (cross-cutting) area/compression Context compression and continuation sessions area/config Config system, migrations, profiles codex comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists provider/openai OpenAI / Codex Responses API provider/xai xAI (Grok) sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Compression effective threshold (~150K) too high for ≤160K-ctx models — never fires, causing cascading 429 on 100K TPM providers

4 participants

@AlexFucuson9 @alt-glitch @teknium1 @tonydwb