AssistantAgent first "think" before calling a tool.

[ekzhu]

When using gpt-4o or similar models with the autogen_agentchat.agents.AssistantAgent, it often just calls the tool directly without first responding with a message that says what tool it will call.

It would be good to show how to either properly prompt the model to make it responds with a "thought" message first before calling tools or build in some mechanism in AssistantAgent that will trigger this behavior.

Second thing I am facing is not being able to get LLM to send message before calling tool.

This is a good idea. Have you tried providing some in-context learning examples in the system message, or through the model_context parameter of AssistantAgent, by adding a few pairs of UserMessage and AssistantMessage in the initial_message field?

Originally posted by @ekzhu in #4886 (comment)

Example usage when add a new parameter think_before_tool_use=True. When this parameter is set to True, the agent will first call the model with tool_choice = "none" and then call the model again with tool_choice="auto".

import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.ui import Console
from autogen_ext.models.openai import OpenAIChatCompletionClient

async def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny."

async def main() -> None:
    agent = AssistantAgent("assistant", model_client=OpenAIChatCompletionClient(model="gpt-4o"), tools=[get_weather], system_message="You are an helpful assistant.", think_before_tool_use=True)
    await Console(agent.run_stream(task="What is the weather in New York?"))

asyncio.run(main())

---------- user ----------
What is the weather in New York?
---------- assistant ----------
I'll get the current weather information for New York. Please hold on for a moment.
[Prompt tokens: 59, Completion tokens: 18]
---------- assistant ----------
[FunctionCall(id='call_akP2ehKPtzHBMY2m1vuPjwTq', arguments='{"city":"New York"}', name='get_weather')]
[Prompt tokens: 81, Completion tokens: 16]
---------- assistant ----------
[FunctionExecutionResult(content='The weather in New York is sunny.', call_id='call_akP2ehKPtzHBMY2m1vuPjwTq')]
---------- assistant ----------
The weather in New York is sunny.
---------- Summary ----------
Number of messages: 5
Finish reason: Maximum number of turns 1 reached.
Total prompt tokens: 140
Total completion tokens: 34
Duration: 1.33 seconds

One caveat is when the task is unrelated to tool use, the agent will produce two messages e.g.:

...
await Console(team.run_stream(task="What is your capability?"))
...

---------- user ----------
What is your capability?
---------- assistant ----------
I'm an AI assistant designed to help with a wide range of tasks including answering questions, providing information, assisting with problem-solving, and interfacing with specific tools or functions to perform tasks like retrieving weather information, among others. I'm here to assist you with whatever you need, within the scope of my capabilities.
[Prompt tokens: 56, Completion tokens: 62]
---------- assistant ----------
Is there anything specific you would like to know or need help with?
[Prompt tokens: 122, Completion tokens: 16]
---------- Summary ----------
Number of messages: 3
Finish reason: Maximum number of turns 1 reached.
Total prompt tokens: 178
Total completion tokens: 78
Duration: 2.96 seconds

The question is how to reconcile this case. Should we include both messages in the final response's chat_message?

[SuMiaoALi]

You can use prompt words to constrain, which is what I do, so that the big model can give an execution plan before taking action

Answer requirements

• Remember, before each action is executed, please clearly inform the user of your execution plan.

I added it like this, it will basically give an execution plan first

[ekzhu]

@SuMiaoALi thanks for the suggestion.

Could you create a PR to add a tip on how to trigger the "first think then execute" through prompting in the tutorial?

autogen/python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/agents.ipynb

Lines 196 to 197 in d99fb23

	"## Using Tools\n",
	"\n",

You can use

```{tip}
Your prompting tip here.
```

[SuMiaoALi]

我的意思是，你可以通过提示词来达到约束LLM的效果，这与代码无关的。
你可以让LLM先给出执行计划，等人类批准后再执行，你也可以让AI给出执行计划仅通知你，它可以随时自主执行。
这是可以用提示词实现的

[ekzhu]

我的意思是，你可以通过提示词来达到约束LLM的效果，这与代码无关的。
你可以让LLM先给出执行计划，等人类批准后再执行，你也可以让AI给出执行计划仅通知你，它可以随时自主执行。
这是可以用提示词实现的

I understand. I meant can you provide an example prompt for a common model, like gpt-4o.

[ekzhu]

Not needed for now as it can either be prompted or extracted from tool call response that contain additional text field.