fix(compression): don't let config context_length auto-lower threshold - #58623
Open
AlexFucuson9 wants to merge 1 commit into
Open
fix(compression): don't let config context_length auto-lower threshold#58623AlexFucuson9 wants to merge 1 commit into
AlexFucuson9 wants to merge 1 commit into
Conversation
auxiliary.compression.context_length is intended to bound the compression model's input size. But the auto-lower logic used it as the model's natural context length, incorrectly lowering the session threshold when the config value is smaller than the threshold. When a user sets context_length=128000 to limit the summarizer's workload on a model with 800K natural context, the auto-lower would reduce the threshold from 400K to 128K — causing compression to fire on every turn with almost no token reduction. Fix: resolve the model's natural context length (without config override) separately, and use only the natural length for auto-lower decisions. The config override still bounds the compression model's input size. Fixes NousResearch#58407
Collaborator
|
Contributor
|
Thanks for isolating the every-turn compression loop; current Problems
Suggested changes
Automated hermes-sweeper review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
auxiliary.compression.context_lengthis intended to bound the compression model's input size. But the auto-lower logic used it as the model's natural context length, incorrectly lowering the session threshold.Problem
When a user sets
auxiliary.compression.context_length: 128000to limit the summarizer's workload on a model with 800K natural context, the auto-lower reduces the threshold from 400K to 128K. This causes compression to fire on every turn with almost no token reduction, burning ~1.3M input tokens on ineffective summarization calls.Root Cause
In
conversation_compression.py, the auto-lower logic does:The
config_context_lengthoverride is returned asaux_contextbyget_model_context_length(), so the auto-lower treats the user's input-size limit as the model's actual context window.Fix
Resolve the model's NATURAL context length (without config override) separately, and use only the natural length for auto-lower decisions. The config override still bounds the compression model's input size.
Fixes #58407