Add new tool call streaming system #1713

404Wolf · 2025-12-01T21:55:16Z

I understand that this repository is auto-generated and my pull request may not be merged

Changes being requested

Update the tool call pattern to use a new async iterable style pattern that makes more easily accessible tool calls as they are fired, and API responses.

Additional context & links

zanllan23-dev

Ok

RobertCraigie · 2025-12-02T18:07:03Z

src/helpers/beta/zod.ts

+import type { infer as zodInfer, ZodType } from 'zod';
+import * as z from 'zod';


nit: these should use v4 imports https://zod.dev/library-authors#how-to-support-zod-3-and-zod-4-simultaneously

RobertCraigie · 2025-12-02T18:11:42Z

src/resources/beta/beta.ts

  chatkit: ChatKitAPI.ChatKit = new ChatKitAPI.ChatKit(this._client);
  assistants: AssistantsAPI.Assistants = new AssistantsAPI.Assistants(this._client);
  threads: ThreadsAPI.Threads = new ThreadsAPI.Threads(this._client);
+  chat: ChatAPI.Chat = new ChatAPI.Chat(this._client);


you should define a sibling chat.ts file in this directory and define a new class there instead of adding toolRunner() to the existing resource class, as now it'll be accessible from client.chat.completions.toolRunner() which isn't what we want yet

I extended and created a BetaChat, I'm not entirely sure if this is what you had in mind, but it limits it to .beta.chat and not just .chat like we want

update types start setting up tests get non streaming working begin work on modifying anthropic tests to openai update models

RobertCraigie · 2025-12-03T19:26:13Z

src/resources/beta/chat.ts

+} from '../../lib/beta/BetaToolRunner';
+import { Chat as ChatResource } from '../../resources';
+
+export class BetaCompletions extends ChatResource.Completions {


this should just extend the base APIResource as there's no benefit to exposing the other methods on Completions here as well

Suggested change

export class BetaCompletions extends ChatResource.Completions {

export class BetaCompletions extends APIResource {

RobertCraigie

Looking good.

Once we've ironed out some of the naming questions, could you also add docs for this?

RobertCraigie · 2025-12-03T19:27:33Z

src/helpers/beta/zod.ts

+ */
+export function betaZodTool<InputSchema extends ZodType>(options: {
+  name: string;
+  inputSchema: InputSchema;


nit: this should be parameters to match the API shape

RobertCraigie · 2025-12-03T19:28:10Z

src/helpers/beta/zod.ts

+  if (jsonSchema.type !== 'object') {
+    throw new Error(`Zod schema for tool "${options.name}" must be an object, but got ${jsonSchema.type}`);
+  }
+
+  // TypeScript doesn't narrow the type after the runtime check, so we need to assert it
+  const objectSchema = jsonSchema as typeof jsonSchema & { type: 'object' };


nit: I don't think we need this check? I think we should be able to just do parameters: objectSchema below?

RobertCraigie · 2025-12-03T19:32:01Z

examples/tool-helpers-advanced.ts

+      },
+    ],
+    tools: [
+      betaZodTool({


thinking through these again, I wonder if we should call these helpers betaZodFunction()? because that feels more accurate with respect to the API.

thoughts?

It's a big mix --

They use "tool_calls" in the actual messages for the list of tool calls, "function" as the type of the tool (or at least the ones we are using), and title their docs page "function calling."

Because we're talking about a type of tool call called a function call, for full transparency I'm actually starting to think we should call it "betaZodFunctionTool" (on their page they also refer user supplied functions as "function tool")

RobertCraigie · 2025-12-03T19:33:30Z

src/lib/beta/BetaToolRunner.ts

+/**
+ * A ToolRunner handles the automatic conversation loop between the assistant and tools.
+ *
+ * A ToolRunner is an async iterable that yields either BetaMessage or BetaMessageStream objects


some incorrect references here 😅

src/lib/beta/BetaToolRunner.ts

RobertCraigie · 2025-12-03T19:42:06Z

src/lib/beta/BetaToolRunner.ts

+                tools: params.tools,
+                messages: params.messages,
+                model: params.model,


q: why are these explicitly listed?

Because typescript can't infer that stream: false even though we're in the else. tools/messages/model are not necessary

In completions.ts I saw we always explicitly provide stream (we don't leave it as undefined) so I think it's okay to override it here to get nicer inference

RobertCraigie · 2025-12-03T19:43:12Z

src/lib/beta/BetaToolRunner.ts

+            // TODO: what if it is empty?
+            if (prevMessage) {


this seems redundant given we check prevMessage above?

I'll remove both checks -- both are no longer relevant under the assumption that the API always will return at least one choice

RobertCraigie · 2025-12-03T19:43:47Z

src/lib/beta/BetaToolRunner.ts

+            );
+
+            this.#message = this.#chatCompletion.then((resp) => resp.choices.at(0)!.message);
+            this.#message.catch(() => {});


q: should we be logging the error? or is this safe to just ignore if the error is already reported somewhere else?

When the caller does an "await" (i.e. for await const of), if it was rejected they will get to see the error since they'll be awaiting a rejected promise (of which we ignored the error previously -- here)

If i'm understanding right, the footgun is when they do iterator.next() without an await or looking at the result then there'll be an uncaught exception that is caught and ignored. In the way people use this I think that that is okay

Also, in the "handles api errors" test we show where you can allow an api error to happen but keep on iterating anyway, which is an artifact of how the exception doesn't throw in the actual iterator

src/lib/beta/BetaToolRunner.ts

RobertCraigie

Thinking through naming more again (sorry), we'll likely want to add a similar helper for the Responses API as well in the future, so we should namespace these helpers in some fashion to make it clear they're for Chat Completions instead of Responses, e.g. betaChatFunction() etc.

Could you review the current naming and suggest some alternatives taking this into account?

src/helpers/beta/zod.ts

src/lib/beta/BetaRunnableTool.ts

src/resources/beta/chat.ts

src/lib/beta/BetaToolRunner.ts

RobertCraigie · 2025-12-04T14:02:26Z

README.md

+
+## Tool Helpers
+
+The SDK makes it easy to create and run [function tools with the chats API](https://platform.openai.com/docs/guides/function-calling). You can use Zod schemas or direct JSON schemas to describe the shape of tool input, and then you can run the tools using the `client.beta.messages.toolRunner` method. This method will automatically handle passing the inputs generated by the model into your tools and providing the results back to the model.


FYI looks like the primary examples in those linked docs are for the Responses API, not Chat Completions.

It's where they explain what a function tool call actually is, but yes it focuses on the responses API. Should we re-explain it here instead and not link to it, since it could cause confusion?

404Wolf · 2025-12-04T18:28:29Z

Thinking through naming more again (sorry), we'll likely want to add a similar helper for the Responses API as well in the future, so we should namespace these helpers in some fashion to make it clear they're for Chat Completions instead of Responses, e.g. betaChatFunction() etc.

Could you review the current naming and suggest some alternatives taking this into account?

Sure, some thoughts (in order of preference):

I looked at both types, and they're very similar. In fact, they're the same, except that function tools for responses api have the literal {type: 'function'}. What if they just return one and then let toolrunner convert it automatically? So you use the same utility for both.
If the helpers are API specific maybe we should put them under the resource. So, for instance, Chat.zodFunctionTool()
Similar: we could scope the helpers. e.g. import { zodHelperTool } from "helpers/beta/chat/tools" and import { zodHelperTool } from "helpers/beta/responses/tools".
Alternatively we could, yes, just do something like betaChatZodFunctionTool, but that sort of sucks. We could shorten functionTool to betaChatZodFunction.

…-streaming

404Wolf added 3 commits December 1, 2025 14:46

add chat to beta

8f8d768

start adding option for new iterator

1260ec6

more progress on iterative tool calls

4a71943

zanllan23-dev approved these changes Dec 1, 2025

View reviewed changes

404Wolf added 3 commits December 1, 2025 19:43

fix type issues

55d2d77

finish port

4864dca

add support for options

8e55b1e

RobertCraigie reviewed Dec 2, 2025

View reviewed changes

404Wolf added 18 commits December 2, 2025 14:22

update to new async iterable pattern

95694b6

update types start setting up tests get non streaming working begin work on modifying anthropic tests to openai update models

fix E2E test

c4f95d4

fix more tests

f2834e9

fix more tests

34421a0

most tests passing

d224d00

add more examples

d1fb561

throw when n>1

1093bd0

update another test

844c5a4

make tool runner beta only

49bbff6

update snapshot

f730423

make executable

337d05e

tests passing

410d410

fix type errors

51b200d

unformat mistakenly formatted jsons

d36648c

make nock dev dep

4a23c9d

more minimal package changes

444f9b0

don't await the promise

afb98d2

use zod v4

4c95b49

404Wolf requested a review from RobertCraigie December 3, 2025 17:09

catch with separate update

5b52aa9

RobertCraigie reviewed Dec 3, 2025

View reviewed changes

404Wolf added 2 commits December 3, 2025 15:42

PR feedback

cdeb4e6

start working on docs

3f96a58

404Wolf force-pushed the add-new-tool-call-streaming branch from 50dbc89 to d5d3818 Compare December 3, 2025 21:48

more readme work

c1dd29b

404Wolf force-pushed the add-new-tool-call-streaming branch from d5d3818 to c1dd29b Compare December 3, 2025 21:55

404Wolf added 2 commits December 3, 2025 17:17

remove object check

5a9cb4b

add image tool

d18bbfb

404Wolf force-pushed the add-new-tool-call-streaming branch from b870a1f to d18bbfb Compare December 3, 2025 22:55

RobertCraigie reviewed Dec 4, 2025

View reviewed changes

404Wolf added 5 commits December 4, 2025 12:14

pass directly

658afef

add test for top level array

9b0b3df

rename BetaRunnableTool

d75a202

Solve more TODOs

3c8bdba

add doc

1066d30

404Wolf added 4 commits December 4, 2025 13:30

improve names

b0f8818

Merge remote-tracking branch 'upstream/master' into add-new-tool-call…

6574f39

…-streaming

remove debugging

8eb6c3d

fix more naming

949d8a0

404Wolf force-pushed the add-new-tool-call-streaming branch from 2db4f0a to 949d8a0 Compare December 4, 2025 18:48

404Wolf changed the title ~~[WIP] Add new tool call streaming system~~ Add new tool call streaming system Dec 5, 2025

404Wolf marked this pull request as ready for review December 5, 2025 14:52

404Wolf requested a review from a team as a code owner December 5, 2025 14:52

404Wolf added 2 commits December 5, 2025 10:00

rename to parameters

20f39a5

allow object

ec31a58

404Wolf force-pushed the add-new-tool-call-streaming branch from 2f06c89 to ec31a58 Compare December 5, 2025 15:30

404Wolf added 3 commits December 5, 2025 13:21

make linter happy

ded0bd1

clean up

ac91ad0

change anys to unknowns

c0a0bb0

		import type { infer as zodInfer, ZodType } from 'zod';
		import * as z from 'zod';

	export class BetaCompletions extends ChatResource.Completions {
	export class BetaCompletions extends APIResource {


		## Tool Helpers

		The SDK makes it easy to create and run [function tools with the chats API](https://platform.openai.com/docs/guides/function-calling). You can use Zod schemas or direct JSON schemas to describe the shape of tool input, and then you can run the tools using the `client.beta.messages.toolRunner` method. This method will automatically handle passing the inputs generated by the model into your tools and providing the results back to the model.

Add new tool call streaming system #1713

Are you sure you want to change the base?

Add new tool call streaming system #1713

Conversation

404Wolf commented Dec 1, 2025

Changes being requested

Additional context & links

Uh oh!

zanllan23-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RobertCraigie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

404Wolf Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RobertCraigie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

404Wolf commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

404Wolf Dec 3, 2025 •

edited

Loading

RobertCraigie left a comment •

edited

Loading