Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
dae3cfe
test(e2e): add validation for tool calls and usage stats
steebchen Aug 29, 2025
662a125
fix(models): remove unsupported fields from Google tool parameters
steebchen Aug 29, 2025
70b28fc
fix(chat): resolve tool call issues in streaming responses
steebchen Aug 29, 2025
100bffb
fix(models): clean tool_calls from messages in OpenAI responses API
steebchen Aug 29, 2025
001bf4b
fix(models): transform tool role messages for OpenAI responses API
steebchen Aug 29, 2025
8713827
fix(chat): correct finish_reason logic for tool call responses
steebchen Aug 29, 2025
adf7189
fix(chat): update finish_reason handling for ZAI responses
steebchen Aug 29, 2025
537c296
Revert "fix(chat): update finish_reason handling for ZAI responses"
steebchen Aug 29, 2025
c0931c3
Revert "fix(chat): correct finish_reason logic for tool call responses"
steebchen Aug 29, 2025
c720c14
Revert "fix(models): transform tool role messages for OpenAI response…
steebchen Aug 29, 2025
e78c186
Revert "fix(models): clean tool_calls from messages in OpenAI respons…
steebchen Aug 29, 2025
b146ff3
fix(chat): handle ZAI finish_reason and tool role issues
steebchen Aug 29, 2025
2183636
Merge branch 'main' into fix/toolcalls
steebchen Aug 30, 2025
09debba
Merge branch 'main' into fix/toolcalls
steebchen Aug 30, 2025
5da0c8c
fix(chat): update tool ID generation logic
steebchen Aug 30, 2025
9f65813
fix(e2e): update test assertions for tool_calls and content verification
steebchen Aug 31, 2025
b9c7bd4
fix(models): handle messages with tool calls in API logic
steebchen Aug 31, 2025
6bbb009
Merge remote-tracking branch 'origin/main' into fix/toolcalls
steebchen Aug 31, 2025
aa5e80b
fix(chat): detect tool calls to select appropriate API
steebchen Aug 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 116 additions & 7 deletions apps/gateway/src/api.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,13 +105,6 @@ const testModels = filteredModels
}

return testCases;
})
.filter((testCase) => {
// Filter by TEST_MODELS if specified
if (!specifiedModels) {
return true;
}
return specifiedModels.includes(testCase.model);
});

const providerModels = filteredModels
Expand Down Expand Up @@ -890,6 +883,122 @@ describe("e2e", () => {
},
);

test.each(toolCallModels)(
"tool calls with result $model",
getTestOptions(),
async ({ model }) => {
const res = await app.request("/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer real-token`,
},
body: JSON.stringify({
model: model,
messages: [
{
role: "system",
content:
"You are Noemi, a thoughtful and clear assistant. Your tone is calm, minimal, and human. You write with intention—never too much, never too little. You avoid clichés, speak simply, and offer helpful, grounded answers. When needed, you ask good questions. You don't try to impress—you aim to clarify. You may use metaphors if they bring clarity, but you stay sharp and sincere. You're here to help the user think clearly and move forward, not to overwhelm or overperform.",
},
{
role: "user",
content: "web search for the best ai notetaker apps!!!!",
},
{
role: "assistant",
content: "",
tool_calls: [
{
id: "toolu_015dgN1nk5Ay12iN8e16XPbs",
type: "function",
function: {
name: "webSearch",
arguments: '{"query":"best AI notetaker apps 2024"}',
},
},
],
},
{
role: "tool",
content:
'{"type":"webSearch","query":"best AI notetaker apps 2024","results":[{"title":"My Deep Dive into 25+ AI Note-Taking Apps (The Brutally ... - Reddit","href":"https://www.reddit.com/r/Zoom/comments/1jtbxkf/my_deep_dive_into_25_ai_notetaking_apps_the/","description":"The Good: Think Obsidian meets Miro. Whiteboard-style interface for connecting notes visually. AI assistant can generate summaries and do ..."},{"title":"The 9 best AI meeting assistants in 2025 - Zapier","href":"https://zapier.com/blog/best-ai-meeting-assistant/","description":"Granola automatically transcribes, summarizes, and analyzes your meetings. It also acts as a live notepad, allowing you to manually jot down ..."},{"title":"The Best AI Tools for Taking Notes in 2025 - PCMag","href":"https://www.pcmag.com/picks/best-ai-tools-taking-notes","description":"The popular note-taking app Notion now has AI tools. Notion AI excels at answering questions about your existing data, generating text from a prompt you give it ..."},{"title":"Top 5 BEST AI Note-Taking Apps (Better than Notion?) - YouTube","href":"https://www.youtube.com/watch?v=wGLd43TkCGc","description":"Voicenotes is a voice‑to‑text powerhouse that transcribes and extracts action items in one tap. · Saner is A distraction‑free workspace built for ..."},{"title":"9 Best AI Note-Taking Apps Built For Your Meetings - Quil\'s AI","href":"https://quil.ai/2024/09/12/9-best-ai-note-taking-apps-built-for-your-meetings/","description":"Quil.ai: The AI Note-taker Built for Recruiting Firms. 2. Notion: Write, Plan, Organize. 3. Jamie AI: The Bot-Free AI Note-taker."}],"timestamp":"2025-08-29T01:20:29.553Z"}',
tool_call_id: "toolu_015dgN1nk5Ay12iN8e16XPbs",
},
],
tools: [
{
type: "function",
function: {
name: "webSearch",
description: "Search the web for information",
parameters: {
type: "object",
properties: {
query: {
type: "string",
description: "Search query",
},
},
required: ["query"],
},
},
},
],
tool_choice: "auto",
}),
});

const json = await res.json();
if (logMode) {
console.log(
"tool calls with empty content response:",
JSON.stringify(json, null, 2),
);
}

// Log error response if status is not 200
if (res.status !== 200) {
console.log(
`Error ${res.status} - tool calls with empty content response:`,
JSON.stringify(json, null, 2),
);
}

expect(res.status).toBe(200);
expect(json).toHaveProperty("choices");
expect(json.choices).toHaveLength(1);
expect(json.choices[0]).toHaveProperty("message");

const message = json.choices[0].message;
expect(message).toHaveProperty("role", "assistant");

// Should have proper content (not empty) as a response to the tool call
expect(message).toHaveProperty("content");
expect(typeof message.content).toBe("string");
expect(message.content.length).toBeGreaterThan(0);

// Should have finish reason as stop (not tool_calls since this is a response)
expect(json.choices[0]).toHaveProperty("finish_reason", "stop");

// Validate logs
const log = await validateLogs();
expect(log.streamed).toBe(false);

// Validate usage
expect(json).toHaveProperty("usage");
expect(json.usage).toHaveProperty("prompt_tokens");
expect(json.usage).toHaveProperty("completion_tokens");
expect(json.usage).toHaveProperty("total_tokens");
expect(typeof json.usage.prompt_tokens).toBe("number");
expect(typeof json.usage.completion_tokens).toBe("number");
expect(typeof json.usage.total_tokens).toBe("number");
expect(json.usage.prompt_tokens).toBeGreaterThan(0);
expect(json.usage.completion_tokens).toBeGreaterThan(0);
expect(json.usage.total_tokens).toBeGreaterThan(0);
},
);

Comment on lines +886 to +1006

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Assert no residual tool_calls in final turn and log

Strengthen the test to ensure the assistant returned an actual answer (no further tool calls) and logs don’t record provider-emitted tool calls in this “with result” path.

Apply:

- const log = await validateLogs();
+ const log = await validateLogs();
  expect(log.streamed).toBe(false);
+ // final turn should be a normal answer, not another tool call
+ expect(json.choices[0].finish_reason).toBe("stop");
+ expect(log.toolResults).toBeNull();

Would you like a streaming variant of this test for parity?

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents
In apps/gateway/src/api.e2e.ts around lines 886 to 1001, strengthen the test by
asserting that the final assistant message contains no residual tool_calls and
that the request/processing logs do not record provider-emitted tool calls:
after obtaining message, add an assertion that message.tool_calls is either
undefined/null or an empty array (and fail if any tool_call entries exist), and
extend the validateLogs() checks to assert the logs object does not contain
provider/tool-call entries (e.g., no providerToolCalls or no log entries with
type "tool_call"); keep assertions consistent with existing log shape returned
by validateLogs().

test.each(
testModels.filter((m) => {
const modelDef = models.find((def) => def.id === m.model);
Expand Down
100 changes: 78 additions & 22 deletions apps/gateway/src/chat/chat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,11 @@ function getProviderTokenFromEnv(usedProvider: Provider): string | undefined {
/**
* Parses response content and metadata from different providers
*/
function parseProviderResponse(usedProvider: Provider, json: any) {
function parseProviderResponse(
usedProvider: Provider,
json: any,
messages: any[] = [],
) {
Comment on lines +230 to +234

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Type strictly: avoid any per repo guideline.

Use BaseMessage[] and unknown for json.

-function parseProviderResponse(
-  usedProvider: Provider,
-  json: any,
-  messages: any[] = [],
-) {
+function parseProviderResponse(
+  usedProvider: Provider,
+  json: unknown,
+  messages: BaseMessage[] = [],
+) {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function parseProviderResponse(
usedProvider: Provider,
json: any,
messages: any[] = [],
) {
function parseProviderResponse(
usedProvider: Provider,
json: unknown,
messages: BaseMessage[] = [],
) {
🤖 Prompt for AI Agents
In apps/gateway/src/chat/chat.ts around lines 230 to 234, the function signature
uses any for json and messages which violates the repo typing guideline; change
the signature to use json: unknown and messages: BaseMessage[] (import or
reference BaseMessage type) and update the function body to safely
narrow/validate json (type guards or runtime checks) before accessing its
properties, and treat messages as BaseMessage[] so any downstream operations use
the strict type; ensure any casts are localized and justified with checks.

let content = null;
let reasoningContent = null;
let finishReason = null;
Expand Down Expand Up @@ -412,9 +416,14 @@ function parseProviderResponse(usedProvider: Provider, json: any) {
toolResults = null;
}

// Status mapping (completed -> stop, but tool_calls if function calls present)
// Status mapping with tool call detection for responses API
if (json.status === "completed") {
finishReason = functionCalls.length > 0 ? "tool_calls" : "stop";
// Check if there are tool calls in the response
if (toolResults && toolResults.length > 0) {
finishReason = "tool_calls";
} else {
finishReason = "stop";
}
} else {
finishReason = json.status;
}
Expand All @@ -436,6 +445,36 @@ function parseProviderResponse(usedProvider: Provider, json: any) {
json.choices?.[0]?.message?.reasoning ||
null;
finishReason = json.choices?.[0]?.finish_reason || null;

// ZAI-specific fix for incorrect finish_reason in tool response scenarios
// Only for models that were failing tests: glm-4.5-airx and glm-4.5-flash
if (
usedProvider === "zai" &&
finishReason === "tool_calls" &&
messages.length > 0
) {
const lastMessage = messages[messages.length - 1];
const modelName = json.model;

// Only apply to specific failing models and only when last message was a tool result
if (
(modelName === "glm-4.5-airx" || modelName === "glm-4.5-flash") &&
lastMessage?.role === "tool"
) {
// Check if the response actually contains new tool calls that should be prevented
const hasNewToolCalls =
json.choices?.[0]?.message?.tool_calls?.length > 0;
if (hasNewToolCalls) {
finishReason = "stop";
// Also update JSON to match
if (json.choices?.[0]) {
json.choices[0].finish_reason = "stop";
delete json.choices[0].message.tool_calls;
}
}
}
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

promptTokens = json.usage?.prompt_tokens || null;
completionTokens = json.usage?.completion_tokens || null;
reasoningTokens = json.usage?.reasoning_tokens || null;
Expand Down Expand Up @@ -643,11 +682,14 @@ function extractToolCallsFromProvider(
},
];
}
// Tool arguments come as content_block_delta
// Tool arguments come as content_block_delta - these don't have a direct ID,
// so we return null and let the streaming logic handle the accumulation
// by finding the matching tool call by content block index
if (data.type === "content_block_delta" && data.delta?.partial_json) {
// Return a partial tool call with the index to help with matching
return [
{
id: data.index ? `tool_${data.index}` : "tool_unknown",
_contentBlockIndex: data.index, // Use this for matching
type: "function",
function: {
name: "",
Expand All @@ -664,8 +706,8 @@ function extractToolCallsFromProvider(
return (
parts
.filter((part: any) => part.functionCall)
.map((part: any) => ({
id: part.functionCall.name + "_" + Date.now(),
.map((part: any, index: number) => ({
id: part.functionCall.name + "_" + Date.now() + "_" + index,
type: "function",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
function: {
name: part.functionCall.name,
Expand Down Expand Up @@ -791,10 +833,11 @@ function transformToOpenAIFormat(
},
finish_reason:
finishReason === "STOP"
? toolResults && toolResults.length > 0
? "stop"
: finishReason === "tool_calls" ||
finishReason === "function_call"
? "tool_calls"
: "stop"
: finishReason?.toLowerCase() || "stop",
: finishReason?.toLowerCase() || "stop",
},
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
],
usage: {
Expand Down Expand Up @@ -1303,10 +1346,6 @@ function transformStreamingChunkToOpenAIFormat(
};
} else if (data.candidates?.[0]?.finishReason) {
const finishReason = data.candidates[0].finishReason;
// Check if there are function calls in this response
const hasFunctionCalls = data.candidates?.[0]?.content?.parts?.some(
(part: any) => part.functionCall,
);
transformedData = {
id: data.responseId || `chatcmpl-${Date.now()}`,
object: "chat.completion.chunk",
Expand All @@ -1320,10 +1359,11 @@ function transformStreamingChunkToOpenAIFormat(
},
finish_reason:
finishReason === "STOP"
? hasFunctionCalls
? "stop"
: finishReason === "tool_calls" ||
finishReason === "function_call"
? "tool_calls"
: "stop"
: finishReason?.toLowerCase() || "stop",
: finishReason?.toLowerCase() || "stop",
},
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
],
usage: data.usageMetadata
Expand Down Expand Up @@ -3407,9 +3447,22 @@ chat.openapi(completions, async (c) => {
}
// Merge tool calls (accumulating function arguments)
for (const newCall of toolCallsChunk) {
const existingCall = streamingToolCalls.find(
(call) => call.id === newCall.id,
);
let existingCall = null;

// For Anthropic content_block_delta events, match by content block index
if (
usedProvider === "anthropic" &&
newCall._contentBlockIndex !== undefined
) {
existingCall =
streamingToolCalls[newCall._contentBlockIndex];
} else {
// For other providers and Anthropic content_block_start, match by ID
existingCall = streamingToolCalls.find(
(call) => call.id === newCall.id,
);
}

if (existingCall) {
// Accumulate function arguments
if (newCall.function?.arguments) {
Expand All @@ -3418,7 +3471,10 @@ chat.openapi(completions, async (c) => {
newCall.function.arguments;
}
} else {
streamingToolCalls.push({ ...newCall });
// Clean up temporary fields and add new tool call
const cleanCall = { ...newCall };
delete cleanCall._contentBlockIndex;
streamingToolCalls.push(cleanCall);
}
}
}
Expand Down Expand Up @@ -3984,7 +4040,7 @@ chat.openapi(completions, async (c) => {
cachedTokens,
toolResults,
images,
} = parseProviderResponse(usedProvider, json);
} = parseProviderResponse(usedProvider, json, messages);

// Debug: Log images found in response
console.log("Gateway - parseProviderResponse extracted images:", images);
Expand Down
36 changes: 30 additions & 6 deletions packages/models/src/provider-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -369,9 +369,25 @@ export async function prepareRequestBody(
case "openai": {
if (supportsReasoning) {
// Transform to responses API format (now supports tools as well)
// Clean messages by removing tool_calls and tool_call_id fields
// Also transform 'tool' role to 'user' role since responses API doesn't support 'tool' role
const cleanedMessages = messages.map((msg) => {
const { tool_calls: _, tool_call_id: __, ...cleanMsg } = msg;

// Transform tool role to user role for responses API compatibility
if (cleanMsg.role === "tool") {
return {
...cleanMsg,
role: "user",
};
}

return cleanMsg;
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

OpenAI Responses: dropping tool_calls/tool_call_id and coercing 'tool' → 'user' likely breaks tool-result turns

Removing tool_calls and tool_call_id and rewriting role "tool" to "user" loses the association between prior function calls and their results for reasoning models routed via the Responses API. The model will not see tool outputs, and follow-up assistant replies may be wrong.

Please confirm the correct, current Responses API shape for passing prior tool calls and tool results. If Responses requires content items, map:

  • assistant+tool_calls → function_call items
  • tool messages → tool_result items referencing the call id

I can provide a concrete mapping once you confirm the expected payload. Would you like me to adjust this block to preserve tool context for Responses?


🌐 Web query:

OpenAI Responses API: How do you include previous tool calls and their results in `input`? Are `role:"tool"` messages supported, or must we send `{"type":"tool_result","tool_call_id":...}` items? Point to the official doc/example.

💡 Result:

Short answer: Use the assistant message with a "tool_calls" entry (id + function info) followed by a separate message with role:"tool", tool_call_id set to that id, and content set to the tool result. You do not send {"type":"tool_result",...} — the expected format is assistant.tool_calls then a role:"tool" message that references tool_call_id. See the official example. [1][2]

Sources:

  • OpenAI Cookbook example showing tool-calling flow (assistant message with tool call, then a tool message with tool_call_id and content). [1]
  • Official docs / examples referenced on OpenAI forums showing the same message shape (assistant.tool_calls → tool role message with tool_call_id → assistant interpreting the result). [2]

Preserve tool_calls and tool-role messages for Responses API
Don’t strip tool_calls/tool_call_id or coerce role:"tool" to "user"—the Responses API expects an assistant message with tool_calls followed by a role:"tool" message referencing that tool_call_id to relay the tool result.
Location: packages/models/src/provider-api.ts lines 372–386

🤖 Prompt for AI Agents
In packages/models/src/provider-api.ts around lines 372–386, the code currently
strips tool_calls and tool_call_id and coerces role:"tool" to "user"; revert
that behaviour so tool_calls and tool_call_id are preserved and role:"tool"
remains unchanged for messages destined to the Responses API. Update the mapping
to remove only unsupported/internal fields (if any) but keep tool_calls and
tool_call_id intact, and do not change the role value; ensure resulting messages
still conform to the Responses API schema (preserve assistant message with
tool_calls followed by the tool message that references tool_call_id).


const responsesBody: any = {
model: usedModel,
input: messages,
input: cleanedMessages,
reasoning: {
effort: reasoning_effort || "medium",
summary: "detailed",
Expand Down Expand Up @@ -593,11 +609,19 @@ export async function prepareRequestBody(
if (tools && tools.length > 0) {
requestBody.tools = [
{
functionDeclarations: tools.map((tool: any) => ({
name: tool.function.name,
description: tool.function.description,
parameters: tool.function.parameters,
})),
functionDeclarations: tools.map((tool: any) => {
// Remove additionalProperties and $schema from parameters as Google doesn't accept them
const {
additionalProperties: _additionalProperties,
$schema: _$schema,
...cleanParameters
} = tool.function.parameters || {};
return {
name: tool.function.name,
description: tool.function.description,
parameters: cleanParameters,
};
}),
Comment on lines +681 to +693

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Sanitize tool parameter schemas deeply (nested $schema/additionalProperties)

Only top-level keys are stripped. Nested objects may still carry these fields, causing Google 400s. Consider a recursive prune before assigning parameters.

Apply this refactor in the mapper:

function stripMeta(o: unknown): unknown {
  if (!o || typeof o !== "object") return o;
  if (Array.isArray(o)) return o.map(stripMeta);
  // shallow clone and recurse
  const { additionalProperties, $schema, ...rest } = o as Record<string, unknown>;
  for (const k of Object.keys(rest)) rest[k] = stripMeta(rest[k]);
  return rest;
}

Then:

- const { additionalProperties: _additionalProperties, $schema: _$schema, ...cleanParameters } = tool.function.parameters || {};
+ const cleanParameters = stripMeta(tool.function.parameters || {});
🤖 Prompt for AI Agents
In packages/models/src/provider-api.ts around lines 612 to 624, the mapper
currently only strips top-level additionalProperties and $schema from
tool.function.parameters which leaves nested occurrences and causes Google 400
errors; implement a recursive sanitizer (e.g., stripMeta) that returns
primitives as-is, maps arrays by recursing, and for objects shallow-clones while
removing additionalProperties and $schema then recursing into each remaining
property; replace the existing shallow destructure with parameters:
stripMeta(tool.function.parameters) so all nested occurrences are removed before
assigning parameters.

},
];
}
Expand Down