[Feature Request]: Dynamically removing tools during agent conversation when the tools are not needed or not useful.

[WebsheetPlugin]

Is your feature request related to a problem? Please describe.

To save tokens or improve GPT responses, it makes sense to remove certain tools when they are no longer needed.

Describe the solution you'd like

autogen.agentchat.unregister_function(
hello_world,
caller=picker,
executor=picker
)

Additional context

No response

[Gr3atWh173]

Can you please provide an example scenario where this will be useful?

[WebsheetPlugin]

Sure, I use this in my other chatbot all the time. Let's say you have diferent tools available for the Agent to choose from, but when a tool fails, because it's impossible to execute, you might want to remove it, save tokens, and let the Agent use other tools instead to complete the job.

Another example is simply removing tools to save tokens. Very often you can assume that certain tools will not be needed when other tools have been already picked. For example, if the Agent picked a weather_info tool, then you can assume that the stock_info tool might be not needed. So you remove it.

[WebsheetPlugin]

Some observations:

1. there is method update_tool_signature which even has "is_remove" flag. But it seems removing the function only from llm_config["tools"] but from function_map

I am not sure this is an issue for execution..

I was able to acomplish this feature myself with this code:

def remove_tool(agent, tool_name):
    """Remove tool from the tool list"""
    tools = agent.llm_config.get("tools")
    tools = [tool for tool in tools if tool.get("function", {}).get("name") != tool_name]
    agent.llm_config["tools"] = tools
    map_ =  agent._function_map
    del map_[tool_name]

[ylchan87]

+1 for this feature request

In my use case, each function correspond to an action that the agent could perform.

As the agent interact with the world,

• new actions become available -> register_function
• old actions could be no longer possible -> unregister_function (but no such thing in autogen)

Example:
LLM agent was put to play a text based adventure game

Agent is first at the pub:
functions it can call:

• walk
• talk
• drink

Agent now walks to forest:
functions it can call:

• walk
• talk
• drink
• chop

[JMLX42]

Since the LLM APIs only support a fixed max number of tools (ex: 128 for OpenAI), it is necessary to be able to provide only the most relevant tools. In order to do this, the user should have the possibility to implement its own "tool pool" with an associated selection strategy.

Two ways to do this:

1. Simple: add AssistantAgent::remove_tools(self, tools: List) and AssistantAgent::add_tools(self, tools: List) .
2. Advanced: implement a ToolPool interface/class.

If that's ok with the maintainer I'll start by implementing 1. then we can design 2.

[ekzhu]

@JMLX42 happy to feature your extension if you would like to work on an experimentation of this idea as a community package. This will help when it comes to designing of the ToolPool feature in the main package. We can have a more precise specs for the feature once we have finished experimenting with this idea.

[JMLX42]

"tool pool" aka "workbench" design proposal issue: #4721