Skip to content

fix(gateway): cache thought_signature with correct ID #1448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 1 commit into from
Jan 14, 2026
Merged

fix(gateway): cache thought_signature with correct ID #1448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 2 additions & 19 deletions apps/gateway/src/chat/tools/extract-tool-calls.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
import { redisClient } from "@llmgateway/cache";
import { logger } from "@llmgateway/logger";

import type { Provider } from "@llmgateway/models";

/**
Expand Down Expand Up @@ -45,6 +42,8 @@ export function extractToolCalls(data: any, provider: Provider): any[] | null {
case "google-vertex": {
// Google AI Studio tool calls in streaming
// Include thoughtSignature if present (required for Gemini 3 multi-turn conversations)
// Note: Redis caching of thought_signature happens in transform-streaming-to-openai.ts
// where the actual tool_call ID sent to clients is generated
const parts = data.candidates?.[0]?.content?.parts || [];
return (
parts
Expand All @@ -65,22 +64,6 @@ export function extractToolCalls(data: any, provider: Provider): any[] | null {
thought_signature: part.thoughtSignature,
},
};
// Cache thoughtSignature in Redis for server-side retrieval in multi-turn conversations
// This is especially important when OpenAI SDKs don't preserve extra_content
redisClient
.setex(
`thought_signature:${toolCall.id}`,
86400, // 1 day expiration
part.thoughtSignature,
)
.catch((err) => {
logger.error(
"Failed to cache thought_signature in streaming",
{
err,
},
);
});
}
return toolCall;
}) || null
Expand Down
22 changes: 21 additions & 1 deletion apps/gateway/src/chat/tools/transform-streaming-to-openai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { redisClient } from "@llmgateway/cache";
import { logger } from "@llmgateway/logger";

import { calculatePromptTokensFromMessages } from "./calculate-prompt-tokens.js";
Expand Down Expand Up @@ -423,8 +424,10 @@ export function transformStreamingToOpenai(

if (part.functionCall) {
const callIndex = toolCalls.length;
const toolCallId =
part.functionCall.name + "_" + Date.now() + "_" + callIndex;
toolCalls.push({
id: part.functionCall.name + "_" + Date.now() + "_" + callIndex,
id: toolCallId,
type: "function",
index: partIndex,
function: {
Expand All @@ -443,6 +446,23 @@ export function transformStreamingToOpenai(
}
: undefined,
});

// Cache thoughtSignature in Redis for server-side retrieval in multi-turn conversations
// This is especially important when OpenAI SDKs don't preserve extra_content/provider_extra
if (sig) {
redisClient
.setex(
`thought_signature:${toolCallId}`,
86400, // 1 day expiration
sig,
)
.catch((err) => {
logger.error(
"Failed to cache thought_signature in streaming transform",
{ err },
);
});
}
}

if (sig) {
Expand Down
Loading