Tool grouping

[MackinnonBuck]

Summary

This PR demonstrates a prototype middleware that allows grouping related tools. Groups are hidden from the inner chat client by default, but a well-known "expand" function can be called to expand one group at a time. By reducing the number of tools available to the model at a time, its ability to select the correct tool improves.

Following is an overview of the changes in this PR:

• Add ToolGroupingChatClient, which provides tool grouping and expansion functionality.
• Introduce AIToolGroup to model stable, named batches of tools.
• Add integration tests validating the feasibility of approach.

Usage example

using Microsoft.Extensions.AI;

var travelUtilities = AIToolGroup.Create(
    "TravelUtilities",
    "Travel planning helpers that generate itinerary tokens.",
    [generateItinerary, summarizePacking]);

var financeUtilities = AIToolGroup.Create(
    name: "FinanceUtilities",
    description: "Budgeting helpers.",
    tools: [budgetTool]));

var options = new ChatOptions
{
    Tools = [travelUtilities, financeUtilities]
};

IChatClient client = baseClient
    .AsBuilder()
    .UseToolGrouping(/* ... */)
    .UseToolReduction(/* ... */) // Optionally further reduce the set of available tools
    .UseFunctionInvocation()
    .Build();

var response = await client.GetResponseAsync("Plan a weekend getaway to Lisbon and include a budget tip.");
// ...

Implementation notes

This middleware contains a lot of the same complexities present in FunctionInvokingChatClient: cloning options, honoring ConversationId, aggregating Usage across inner calls, and replaying tool-result messages into the history. The expansion loop is deliberately limited to one active group per top-level request to prevent degradation of tool selection as the number of expanded groups increases.

Microsoft Reviewers: Open in CodeFlow

[ericstj]

This is very clever, I had a couple small questions on behavior.

[ericstj]

So we only group tools known at initialization? What about those added per call to GetResponse? I could imagine some sort of callback to assign a group. Or we could let folks annotate a tool with a group and store that in AdditionalProperties.

[ericstj]

Another thought is auto-grouping based on similarity of descriptions / embedding.

[MackinnonBuck]

That's definitely something to consider. We could have a way to annotate a tool with a "category", similar to how tool descriptions are defined. Then chat clients like ToolGroupingChatClient can utilize the category to perform grouping on a per-response basis. This keeps the tool configuration completely in ChatOptions rather than having it be split between ChatOptions and other middleware.

Or maybe we could change the AIToolGroup type to extend AITool. That way you could put tool groups directly in the ChatOptions. You could even make AIToolGroup unsealed and have a virtual GetToolsAsync() method that can be overridden if you want to dynamically expose different sets of tools based on some other logic (this could be used to implement the "get more tools" function you mentioned in your other comment). I'll think about this a bit more.

[MackinnonBuck]

Some notes:

• We probably need to update FunctionInvokingChatClient to handle tool groups (likely just expanding all the tools in all groups).
• Instead of ToolGroupingChatClient, we could consider calling it ToolCollapsingChatClient or ToolExpandingChatClient.
• We could also consider having some kind of automatic grouping mechanism so that the developer gets the benefits of tool groups without having to define them manually. This could be done as a follow-up.

[ericstj]

We might also want to note that the FunctionInvokingChatClient will add additional tools through it's AdditionalTools property.

We may want t support a similar property on this client as well as any associated API around it if folks want grouped AdditionalTools

[MackinnonBuck]

My understanding of AdditionalTools is that it actually won't change which options are sent through to the inner chat client via ChatOptions.Tools. It's present so that if a service has a set of built-in tools that don't need to be sent on every request, those can be handled by FunctionInvokingChatClient.

With that in mind, I'm not totally sure how helpful it would be to support selecting tool groups in AdditionalProperties for expansion. The groups would be invisible to the AI service, so it wouldn't know how to expand them, right?

I did just push an update that changes FunctionInvokingChatClient to expand any tool groups that it receives, including any present in AdditionalProperties. This allows tool groups to be used as a way to dynamically select which tools get exposed, even if ToolGroupingChatClient isn't used.

Let me know if I'm not understanding the suggestion! 🙂

[ericstj]

I see, I hadn't noticed that those wouldn't be sent to the chat client, however you're right. I guess I don't fully understand the purpose of AdditionalTools if hte model isn't aware of them. I had thought it was just a scoping thing - where tools could be specified on the instance or on the call, but that doesn't seem to be the case. I agree you don't need to be worried about them if they never are exposed to the underlying client.

[stephentoub]

I guess I don't fully understand the purpose of AdditionalTools if hte model isn't aware of them.

It's for cases where the service is already aware of them, eg when you create an OpenAI Assistant and configure it with various tools. The service is aware of those tools and the model can request those tools, even though you haven't sent them as part of the request.