Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AgentChat tutorial update to include model context usage and langchain tool #4843

Merged
merged 12 commits into from
Dec 30, 2024
Merged

AgentChat tutorial update to include model context usage and langchain tool #4843

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8d5a3c8
Doc update to include model context usage
ekzhu Dec 28, 2024
81fceae
add langchain tools
ekzhu Dec 29, 2024
ba3e47e
update langchain tool wrapper api doc
ekzhu Dec 29, 2024
40ee265
updat
ekzhu Dec 29, 2024
6f3b074
update
ekzhu Dec 29, 2024
5cd1229
format
ekzhu Dec 29, 2024
fba254b
add langchain experimental dev dep
ekzhu Dec 29, 2024
03d2937
type
ekzhu Dec 29, 2024
197e362
Fix type
ekzhu Dec 29, 2024
a8dcbc1
Fix some types in langchain adapter
ekzhu Dec 29, 2024
7c738f5
type ignores
ekzhu Dec 30, 2024
20d6d06
Merge branch 'main' into model-context-doc
ekzhu Dec 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 18 additions & 11 deletions python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,40 +72,47 @@ class AssistantAgent(BaseChatAgent):
the inner messages as they are created, and the :class:`~autogen_agentchat.base.Response`
object as the last item before closing the generator.
.. note::
The caller must only pass the new messages to the agent on each call
to the :meth:`on_messages` or :meth:`on_messages_stream` method.
The agent maintains its state between calls to these methods.
Do not pass the entire conversation history to the agent on each call.
.. note::
The assistant agent is not thread-safe or coroutine-safe.
It should not be shared between multiple tasks or coroutines, and it should
not call its methods concurrently.
Tool call behavior:
* If the model returns no tool call, then the response is immediately returned as a :class:`~autogen_agentchat.messages.TextMessage` in :attr:`~autogen_agentchat.base.Response.chat_message`.
* When the model returns tool calls, they will be executed right away:
- When `reflect_on_tool_use` is False (default), the tool call results are returned as a :class:`~autogen_agentchat.messages.ToolCallSummaryMessage` in :attr:`~autogen_agentchat.base.Response.chat_message`. `tool_call_summary_format` can be used to customize the tool call summary.
- When `reflect_on_tool_use` is True, the another model inference is made using the tool calls and results, and the text response is returned as a :class:`~autogen_agentchat.messages.TextMessage` in :attr:`~autogen_agentchat.base.Response.chat_message`.
.. note::
By default, the tool call results are returned as response when tool calls are made.
So it is recommended to pay attention to the formatting of the tools return values,
especially if another agent is expecting them in a specific format.
Use `tool_call_summary_format` to customize the tool call summary, if needed.
Hand off behavior:
* If a handoff is triggered, a :class:`~autogen_agentchat.messages.HandoffMessage` will be returned in :attr:`~autogen_agentchat.base.Response.chat_message`.
* If there are tool calls, they will also be executed right away before returning the handoff.
.. note::
The assistant agent is not thread-safe or coroutine-safe.
It should not be shared between multiple tasks or coroutines, and it should
not call its methods concurrently.
If multiple handoffs are detected, only the first handoff is executed.
.. note::
By default, the tool call results are returned as response when tool calls are made.
So it is recommended to pay attention to the formatting of the tools return values,
especially if another agent is expecting them in a specific format.
Use `tool_call_summary_format` to customize the tool call summary, if needed.
.. note::
If multiple handoffs are detected, only the first handoff is executed.
Limit context size sent to the model:
You can limit the number of messages sent to the model by setting
the `model_context` parameter to a :class:`~autogen_core.model_context.BufferedChatCompletionContext`.
This will limit the number of recent messages sent to the model and can be useful
when the model has a limit on the number of tokens it can process.
Args:
Expand Down
193 changes: 154 additions & 39 deletions python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/agents.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from autogen_agentchat.agents import AssistantAgent\n",
"from autogen_agentchat.messages import TextMessage\n",
"from autogen_agentchat.ui import Console\n",
"from autogen_core import CancellationToken\n",
"from autogen_ext.models.openai import OpenAIChatCompletionClient\n",
"\n",
Expand Down Expand Up @@ -113,42 +114,13 @@
"```{note}\n",
"Unlike in v0.2 AgentChat, the tools are executed by the same agent directly within\n",
"the same call to {py:meth}`~autogen_agentchat.agents.AssistantAgent.on_messages`.\n",
"```\n",
"\n",
"## User Proxy Agent\n",
"\n",
"{py:class}`~autogen_agentchat.agents.UserProxyAgent` is a built-in agent that\n",
"provides one way for a user to intervene in the process. This agent will put the team in a temporary blocking state, and thus any exceptions or runtime failures while in the blocked state will result in a deadlock. It is strongly advised that this agent be coupled with a timeout mechanic and that all errors and exceptions emanating from it are handled."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from autogen_agentchat.agents import UserProxyAgent\n",
"\n",
"\n",
"async def user_proxy_run() -> None:\n",
" user_proxy_agent = UserProxyAgent(\"user_proxy\")\n",
" response = await user_proxy_agent.on_messages(\n",
" [TextMessage(content=\"What is your name? \", source=\"user\")], cancellation_token=CancellationToken()\n",
" )\n",
" print(f\"Your name is {response.chat_message.content}\")\n",
"\n",
"\n",
"# Use asyncio.run(user_proxy_run()) when running in a script.\n",
"await user_proxy_run()"
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The User Proxy agent is ideally used for on-demand human-in-the-loop interactions for scenarios such as Just In Time approvals, human feedback, alerts, etc. For slower user interactions, consider terminating a team using a termination condition and start another one from\n",
"{py:meth}`~autogen_agentchat.base.TaskRunner.run` or {py:meth}`~autogen_agentchat.base.TaskRunner.run_stream` with another message.\n",
"\n",
"## Streaming Messages\n",
"\n",
"We can also stream each message as it is generated by the agent by using the\n",
Expand Down Expand Up @@ -183,9 +155,6 @@
}
],
"source": [
"from autogen_agentchat.ui import Console\n",
"\n",
"\n",
"async def assistant_run_stream() -> None:\n",
" # Option 1: read each message from the stream (as shown in the previous example).\n",
" # async for message in agent.on_messages_stream(\n",
Expand Down Expand Up @@ -216,15 +185,161 @@
"with the final item being the response message in the {py:attr}`~autogen_agentchat.base.Response.chat_message` attribute.\n",
"\n",
"From the messages, you can observe that the assistant agent utilized the `web_search` tool to\n",
"gather information and responded based on the search results.\n",
"gather information and responded based on the search results."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using Tools\n",
"\n",
"## Understanding Tool Calling\n",
"Large Language Models (LLMs) are typically limited to generating text or code responses. \n",
"However, many complex tasks benefit from the ability to use external tools that perform specific actions,\n",
"such as fetching data from APIs or databases.\n",
"\n",
"Large Language Models (LLMs) are typically limited to generating text or code responses. However, many complex tasks benefit from the ability to use external tools that perform specific actions, such as fetching data from APIs or databases.\n",
"To address this limitation, modern LLMs can now accept a list of available tool schemas \n",
"(descriptions of tools and their arguments) and generate a tool call message. \n",
"This capability is known as **Tool Calling** or **Function Calling** and \n",
"is becoming a popular pattern in building intelligent agent-based applications.\n",
"Refer to the documentation from [OpenAI](https://platform.openai.com/docs/guides/function-calling) \n",
"and [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) for more information about tool calling in LLMs.\n",
"\n",
"To address this limitation, modern LLMs can now accept a list of available tool schemas (descriptions of tools and their arguments) and generate a tool call message. This capability is known as **Tool Calling** or **Function Calling** and is becoming a popular pattern in building intelligent agent-based applications.\n",
"In AgentChat, the assistant agent can use tools to perform specific actions.\n",
"The `web_search` tool is one such tool that allows the assistant agent to search the web for information.\n",
"A custom tool can be a Python function or a subclass of the {py:class}`~autogen_core.tools.BaseTool`.\n",
"\n",
"For more information on tool calling, refer to the documentation from [OpenAI](https://platform.openai.com/docs/guides/function-calling) and [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/tool-use)."
"### Langchain Tools\n",
"\n",
"In addition to custom tools, you can also use tools from the Langchain library\n",
"by wrapping them in {py:class}`~autogen_ext.tools.langchain.LangChainToolAdapter`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"---------- assistant ----------\n",
"[FunctionCall(id='call_BEYRkf53nBS1G2uG60wHP0zf', arguments='{\"query\":\"df[\\'Age\\'].mean()\"}', name='python_repl_ast')]\n",
"[Prompt tokens: 111, Completion tokens: 22]\n",
"---------- assistant ----------\n",
"[FunctionExecutionResult(content='29.69911764705882', call_id='call_BEYRkf53nBS1G2uG60wHP0zf')]\n",
"---------- assistant ----------\n",
"29.69911764705882\n",
"---------- Summary ----------\n",
"Number of inner messages: 2\n",
"Total prompt tokens: 111\n",
"Total completion tokens: 22\n",
"Duration: 0.62 seconds\n"
]
},
{
"data": {
"text/plain": [
"Response(chat_message=ToolCallSummaryMessage(source='assistant', models_usage=None, content='29.69911764705882', type='ToolCallSummaryMessage'), inner_messages=[ToolCallRequestEvent(source='assistant', models_usage=RequestUsage(prompt_tokens=111, completion_tokens=22), content=[FunctionCall(id='call_BEYRkf53nBS1G2uG60wHP0zf', arguments='{\"query\":\"df[\\'Age\\'].mean()\"}', name='python_repl_ast')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='assistant', models_usage=None, content=[FunctionExecutionResult(content='29.69911764705882', call_id='call_BEYRkf53nBS1G2uG60wHP0zf')], type='ToolCallExecutionEvent')])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"from autogen_ext.tools.langchain import LangChainToolAdapter\n",
"from langchain_experimental.tools.python.tool import PythonAstREPLTool\n",
"\n",
"df = pd.read_csv(\"https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv\")\n",
"tool = LangChainToolAdapter(PythonAstREPLTool(locals={\"df\": df}))\n",
"model_client = OpenAIChatCompletionClient(model=\"gpt-4o\")\n",
"agent = AssistantAgent(\n",
" \"assistant\", tools=[tool], model_client=model_client, system_message=\"Use the `df` variable to access the dataset.\"\n",
")\n",
"await Console(\n",
" agent.on_messages_stream(\n",
" [TextMessage(content=\"What's the average age of the passengers?\", source=\"user\")], CancellationToken()\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using Model Context\n",
"\n",
"{py:class}`~autogen_agentchat.agents.AssistantAgent` has a `model_context`\n",
"parameter that can be used to pass in a {py:class}`~autogen_core.model_context.ChatCompletionContext`\n",
"object. This allows the agent to use different model contexts, such as\n",
"{py:class}`~autogen_core.model_context.BufferedChatCompletionContext` to\n",
"limit the context sent to the model.\n",
"\n",
"By default, {py:class}`~autogen_agentchat.agents.AssistantAgent` uses\n",
"the {py:class}`~autogen_core.model_context.UnboundedChatCompletionContext`\n",
"which sends the full conversation history to the model. To limit the context\n",
"to the last `n` messages, you can use the {py:class}`~autogen_core.model_context.BufferedChatCompletionContext`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from autogen_core.model_context import BufferedChatCompletionContext\n",
"\n",
"# Create an agent that uses only the last 5 messages in the context to generate responses.\n",
"agent = AssistantAgent(\n",
" name=\"assistant\",\n",
" model_client=model_client,\n",
" tools=[web_search],\n",
" system_message=\"Use tools to solve tasks.\",\n",
" model_context=BufferedChatCompletionContext(buffer_size=5), # Only use the last 5 messages in the context.\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## User Proxy Agent\n",
"\n",
"{py:class}`~autogen_agentchat.agents.UserProxyAgent` is a built-in agent that\n",
"provides one way for a user to intervene in the process. This agent will put the team in a temporary blocking state, and thus any exceptions or runtime failures while in the blocked state will result in a deadlock. It is strongly advised that this agent be coupled with a timeout mechanic and that all errors and exceptions emanating from it are handled."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from autogen_agentchat.agents import UserProxyAgent\n",
"\n",
"\n",
"async def user_proxy_run() -> None:\n",
" user_proxy_agent = UserProxyAgent(\"user_proxy\")\n",
" response = await user_proxy_agent.on_messages(\n",
" [TextMessage(content=\"What is your name? \", source=\"user\")], cancellation_token=CancellationToken()\n",
" )\n",
" print(f\"Your name is {response.chat_message.content}\")\n",
"\n",
"\n",
"# Use asyncio.run(user_proxy_run()) when running in a script.\n",
"await user_proxy_run()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The User Proxy agent is ideally used for on-demand human-in-the-loop interactions for scenarios such as Just In Time approvals, human feedback, alerts, etc. For slower user interactions, consider terminating a team using a termination condition and start another one from\n",
"{py:meth}`~autogen_agentchat.base.TaskRunner.run` or {py:meth}`~autogen_agentchat.base.TaskRunner.run_stream` with another message."
]
},
{
Expand Down
3 changes: 2 additions & 1 deletion python/packages/autogen-ext/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ packages = ["src/autogen_ext"]

[dependency-groups]
dev = [
"autogen_test_utils"
"autogen_test_utils",
"langchain-experimental",
]

[tool.ruff]
Expand Down
48 changes: 43 additions & 5 deletions python/packages/autogen-ext/src/autogen_ext/tools/langchain/_langchain_adapter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from pydantic import BaseModel, Field, create_model

if TYPE_CHECKING:
from langchain_core.tools import Tool as LangChainTool
from langchain_core.tools import BaseTool as LangChainTool


class LangChainToolAdapter(BaseTool[BaseModel, Any]):
Expand All @@ -22,6 +22,44 @@ class LangChainToolAdapter(BaseTool[BaseModel, Any]):
Args:
langchain_tool (LangChainTool): A LangChain tool to wrap
Examples:
Use the `PythonAstREPLTool` from the `langchain_experimental` package to
create a tool that allows you to interact with a Pandas DataFrame.
.. code-block:: python
import asyncio
import pandas as pd
from langchain_experimental.tools.python.tool import PythonAstREPLTool
from autogen_ext.tools.langchain import LangChainToolAdapter
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_agentchat.messages import TextMessage
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.ui import Console
from autogen_core import CancellationToken
async def main() -> None:
df = pd.read_csv("https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv") # type: ignore
tool = LangChainToolAdapter(PythonAstREPLTool(locals={"df": df}))
model_client = OpenAIChatCompletionClient(model="gpt-4o")
agent = AssistantAgent(
"assistant",
tools=[tool],
model_client=model_client,
system_message="Use the `df` variable to access the dataset.",
)
await Console(
agent.on_messages_stream(
[TextMessage(content="What's the average age of the passengers?", source="user")], CancellationToken()
)
)
asyncio.run(main())
"""

def __init__(self, langchain_tool: LangChainTool):
Expand All @@ -32,10 +70,10 @@ def __init__(self, langchain_tool: LangChainTool):
description = self._langchain_tool.description or ""

# Determine the callable method
if hasattr(self._langchain_tool, "func") and callable(self._langchain_tool.func):
assert self._langchain_tool.func is not None
self._callable: Callable[..., Any] = self._langchain_tool.func
elif hasattr(self._langchain_tool, "_run") and callable(self._langchain_tool._run): # pyright: ignore
if hasattr(self._langchain_tool, "func") and callable(self._langchain_tool.func): # type: ignore
assert self._langchain_tool.func is not None # type: ignore
self._callable: Callable[..., Any] = self._langchain_tool.func # type: ignore
elif hasattr(self._langchain_tool, "_run") and callable(self._langchain_tool._run): # type: ignore
self._callable: Callable[..., Any] = self._langchain_tool._run # type: ignore
else:
raise AttributeError(
Expand Down
Loading
Loading