Repository navigation
fix(core): keep token escalation warm across agent rounds #5062
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -208,6 +208,8 @@ export type StreamEvent = | |
| * fresh restart (escalation). The UI should keep the accumulated text | ||
| * buffer so the continuation appends to it. */ | ||
| isContinuation?: boolean; | ||
| /** Set when the retry raised the automatic max output token limit. */ | ||
| maxOutputTokensEscalated?: number; | ||
| } | ||
| | { type: StreamEventType.COMPRESSED; info: ChatCompressionInfo }; | ||
|
|
||
|
|
@@ -2202,19 +2204,29 @@ export class GeminiChat { | |
| // models. | ||
| // Placed outside the retry loop so that any errors from the | ||
| // escalated stream propagate directly (not caught by retry logic). | ||
| const requestedMaxOutputTokens = params.config?.maxOutputTokens; | ||
| const escalatedLimit = Math.max( | ||
| ESCALATED_MAX_TOKENS, | ||
| tokenLimit(model, 'output'), | ||
| ); | ||
| const shouldEscalateMaxOutputTokens = | ||
| requestedMaxOutputTokens === undefined || | ||
| requestedMaxOutputTokens < escalatedLimit; | ||
|
|
||
| if ( | ||
| lastError === null && | ||
| lastFinishReason === FinishReason.MAX_TOKENS && | ||
| !maxTokensEscalated && | ||
| !hasUserMaxTokensOverride | ||
| !hasUserMaxTokensOverride && | ||
| shouldEscalateMaxOutputTokens | ||
| ) { | ||
| maxTokensEscalated = true; | ||
| const escalatedLimit = Math.max( | ||
| ESCALATED_MAX_TOKENS, | ||
| tokenLimit(model, 'output'), | ||
| ); | ||
| const startingLimitLabel = | ||
| requestedMaxOutputTokens === undefined | ||
| ? 'capped default' | ||
| : `${requestedMaxOutputTokens} tokens`; | ||
| debugLogger.info( | ||
| `Output truncated at capped default. Escalating to ${escalatedLimit} tokens.`, | ||
| `Output truncated at ${startingLimitLabel}. Escalating to ${escalatedLimit} tokens.`, | ||
| ); | ||
| // Remove partial model response from history | ||
| // (processStreamResponse already pushed it) | ||
|
|
@@ -2225,7 +2237,10 @@ export class GeminiChat { | |
| self.history.pop(); | ||
| } | ||
| // Signal UI to discard partial output | ||
| yield { type: StreamEventType.RETRY }; | ||
| yield { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] Misleading log message when sticky escalation is active The log at line 2219 reads Consider including the actual starting limit: debugLogger.info(
`Output truncated at ${params.config?.maxOutputTokens ?? 'capped default'}. Escalating to ${escalatedLimit} tokens.`,
);Or skip the log + escalation entirely when no real escalation would occur (i.e., — qwen3.7-max via Qwen Code /review |
||
| type: StreamEventType.RETRY, | ||
| maxOutputTokensEscalated: escalatedLimit, | ||
| }; | ||
| // Retry with escalated max_tokens | ||
| const escalatedParams: SendMessageParameters = { | ||
| ...params, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Suggestion] Redundant re-escalation on subsequent truncated rounds
hasUserMaxTokensOverrideingeminiChat.ts(line 1889) only checkscgConfig.samplingParams.max_tokensandQWEN_CODE_MAX_OUTPUT_TOKENS— it does not seeparams.config.maxOutputTokens. When this sticky value flows through to a subsequent agent round and the model still hitsMAX_TOKENSat the already-escalated limit, GeminiChat's inner escalation block fires again:maxTokensEscalatedresets per call,hasUserMaxTokensOverridestaysfalse, so all guards pass. The escalation computesMath.max(ESCALATED_MAX_TOKENS, tokenLimit(model, 'output'))— the identical value — pops the partial response from history, yields a spuriousRETRY, and makes a duplicate API call at the same limit.This wastes one API call (with full request history) per subsequent truncated round. In a long agent session with many truncation-heavy rounds, the cost compounds.
Fix options:
hasUserMaxTokensOverrideto also checkparams.config?.maxOutputTokens !== undefined, or&& (params.config?.maxOutputTokens ?? 0) < escalatedLimit— qwen3.7-max via Qwen Code /review