Skip to content

fix(chat): improve reasoning token estimation logic #1155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 2 commits into from
Nov 15, 2025
Merged

fix(chat): improve reasoning token estimation logic #1155

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
baf60fb
fix(chat): improve reasoning token estimation logic
steebchen Nov 15, 2025
2e97acc
fix(chat): simplify reasoning token calculation logic
steebchen Nov 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions apps/gateway/src/chat/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2914,6 +2914,21 @@ chat.openapi(completions, async (c) => {
(calculatedPromptTokens || 0) + (calculatedCompletionTokens || 0);
}

// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (!reasoningTokens && fullReasoningContent) {
try {
calculatedReasoningTokens = encode(fullReasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text in streaming",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens =
estimateTokensFromContent(fullReasoningContent);
}
}
Comment on lines +2917 to +2931

@coderabbitai coderabbitai Bot Nov 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use explicit null check to preserve provider-supplied zero values.

The condition !reasoningTokens evaluates to true when reasoningTokens is 0, which means the code will recalculate even if a provider explicitly returns 0 to indicate no reasoning tokens were used.

Apply this diff to use a more explicit check:

 // Estimate reasoning tokens if not provided but reasoning content exists
 let calculatedReasoningTokens = reasoningTokens;
-if (!reasoningTokens && fullReasoningContent) {
+if (reasoningTokens == null && fullReasoningContent) {
   try {
     calculatedReasoningTokens = encode(fullReasoningContent).length;
   } catch (error) {
     // Fallback to simple estimation if encoding fails
     logger.error(
       "Failed to encode reasoning text in streaming",
       error instanceof Error ? error : new Error(String(error)),
     );
     calculatedReasoningTokens =
       estimateTokensFromContent(fullReasoningContent);
   }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (!reasoningTokens && fullReasoningContent) {
try {
calculatedReasoningTokens = encode(fullReasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text in streaming",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens =
estimateTokensFromContent(fullReasoningContent);
}
}
// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (reasoningTokens == null && fullReasoningContent) {
try {
calculatedReasoningTokens = encode(fullReasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text in streaming",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens =
estimateTokensFromContent(fullReasoningContent);
}
}
🤖 Prompt for AI Agents 
In apps/gateway/src/chat/chat.ts around lines 2917 to 2931, the check `if
(!reasoningTokens && fullReasoningContent)` treats a provider-supplied 0 as
missing and will recalculate; change the condition to explicitly test for
null/undefined (e.g., reasoningTokens === null || reasoningTokens === undefined)
combined with fullReasoningContent so that an explicit zero is preserved, then
proceed with the existing try/catch encoding and fallback estimation logic.

// Check if the response finished successfully but has no content, tokens, or tool calls
// This indicates an empty response which should be marked as an error
// Do this check BEFORE sending usage chunks to ensure proper event ordering
Expand Down Expand Up @@ -3129,7 +3144,7 @@ chat.openapi(completions, async (c) => {
promptTokens: calculatedPromptTokens?.toString() || null,
completionTokens: calculatedCompletionTokens?.toString() || null,
totalTokens: calculatedTotalTokens?.toString() || null,
reasoningTokens: reasoningTokens,
reasoningTokens: calculatedReasoningTokens?.toString() || null,
cachedTokens: cachedTokens?.toString() || null,
hasError: streamingError !== null,
errorDetails: streamingError
Expand Down Expand Up @@ -3578,6 +3593,20 @@ chat.openapi(completions, async (c) => {
completionTokens,
);

// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (!reasoningTokens && reasoningContent) {
try {
calculatedReasoningTokens = encode(reasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens = estimateTokensFromContent(reasoningContent);
}
}
Comment on lines +3596 to +3609

@coderabbitai coderabbitai Bot Nov 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use explicit null check to preserve provider-supplied zero values.

Same issue as in the streaming path: the condition !reasoningTokens will be true when reasoningTokens is 0, causing recalculation even when a provider explicitly returns 0.

Apply this diff:

 // Estimate reasoning tokens if not provided but reasoning content exists
 let calculatedReasoningTokens = reasoningTokens;
-if (!reasoningTokens && reasoningContent) {
+if (reasoningTokens == null && reasoningContent) {
   try {
     calculatedReasoningTokens = encode(reasoningContent).length;
   } catch (error) {
     // Fallback to simple estimation if encoding fails
     logger.error(
       "Failed to encode reasoning text",
       error instanceof Error ? error : new Error(String(error)),
     );
     calculatedReasoningTokens = estimateTokensFromContent(reasoningContent);
   }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (!reasoningTokens && reasoningContent) {
try {
calculatedReasoningTokens = encode(reasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens = estimateTokensFromContent(reasoningContent);
}
}
// Estimate reasoning tokens if not provided but reasoning content exists
let calculatedReasoningTokens = reasoningTokens;
if (reasoningTokens == null && reasoningContent) {
try {
calculatedReasoningTokens = encode(reasoningContent).length;
} catch (error) {
// Fallback to simple estimation if encoding fails
logger.error(
"Failed to encode reasoning text",
error instanceof Error ? error : new Error(String(error)),
);
calculatedReasoningTokens = estimateTokensFromContent(reasoningContent);
}
}
🤖 Prompt for AI Agents 
In apps/gateway/src/chat/chat.ts around lines 3596 to 3609, the code uses a
falsy check (`!reasoningTokens`) which treats a valid provider-supplied zero as
missing; change the condition to an explicit null/undefined check (e.g.,
`reasoningTokens == null` or `reasoningTokens === undefined || reasoningTokens
=== null`) so zero is preserved, keep the existing try/catch and fallback logic
intact, and mirror the same explicit check used in the streaming path if
present.

const costs = calculateCosts(
usedModel,
usedProvider,
Expand Down Expand Up @@ -3679,7 +3708,7 @@ chat.openapi(completions, async (c) => {
(
(calculatedPromptTokens || 0) + (calculatedCompletionTokens || 0)
).toString(),
reasoningTokens: reasoningTokens,
reasoningTokens: calculatedReasoningTokens?.toString() || null,
cachedTokens: cachedTokens?.toString() || null,
hasError: hasEmptyNonStreamingResponse,
streamed: false,
Expand Down
Loading