Fig.1 illustrates the general flow of AgentEval
- -**TL;DR:** -* As a developer of an LLM-powered application, how can you assess the utility it brings to end users while helping them with their tasks? -* To shed light on the question above, we introduce `AgentEval` — the first version of the framework to assess the utility of any LLM-powered application crafted to assist users in specific tasks. AgentEval aims to simplify the evaluation process by automatically proposing a set of criteria tailored to the unique purpose of your application. This allows for a comprehensive assessment, quantifying the utility of your application against the suggested criteria. -* We demonstrate how `AgentEval` work using [math problems dataset](https://microsoft.github.io/autogen/blog/2023/06/28/MathChat) as an example in the [following notebook](https://github.com/microsoft/autogen/blob/main/notebook/agenteval_cq_math.ipynb). Any feedback would be useful for future development. Please contact us on our [Discord](http://aka.ms/autogen-dc). - - -## Introduction - - AutoGen aims to simplify the development of LLM-powered multi-agent systems for various applications, ultimately making end users' lives easier by assisting with their tasks. Next, we all yearn to understand how our developed systems perform, their utility for users, and, perhaps most crucially, how we can enhance them. Directly evaluating multi-agent systems poses challenges as current approaches predominantly rely on success metrics – essentially, whether the agent accomplishes tasks. However, comprehending user interaction with a system involves far more than success alone. Take math problems, for instance; it's not merely about the agent solving the problem. Equally significant is its ability to convey solutions based on various criteria, including completeness, conciseness, and the clarity of the provided explanation. Furthermore, success isn't always clearly defined for every task. - - Rapid advances in LLMs and multi-agent systems have brought forth many emerging capabilities that we're keen on translating into tangible utilities for end users. We introduce the first version of `AgentEval` framework - a tool crafted to empower developers in swiftly gauging the utility of LLM-powered applications designed to help end users accomplish the desired task. - - - -Fig. 2 provides an overview of the tasks taxonomy
- - -Let's first look into an overview of the suggested task taxonomy that a multi-agent system can be designed for. In general, the tasks can be split into two types, where: -* _Success is not clearly defined_ - refer to instances when users utilize a system in an assistive manner, seeking suggestions rather than expecting the system to solve the task. For example, a user might request the system to generate an email. In many cases, this generated content serves as a template that the user will later edit. However, defining success precisely for such tasks is relatively complex. -* _Success is clearly defined_ - refer to instances where we can clearly define whether a system solved the task or not. Consider agents that assist in accomplishing household tasks, where the definition of success is clear and measurable. This category can be further divided into two separate subcategories: - * _The optimal solution exits_ - these are tasks where only one solution is possible. For example, if you ask your assistant to turn on the light, the success of this task is clearly defined, and there is only one way to accomplish it. - * _Multiple solutions exist_ - increasingly, we observe situations where multiple trajectories of agent behavior can lead to either success or failure. In such cases, it is crucial to differentiate between the various successful and unsuccessful trajectories. For example, when you ask the agent to suggest you a food recipe or tell you a joke. - -In our `AgentEval` framework, we are currently focusing on tasks where _Success is clearly defined_. Next, we will introduce the suggested framework. - -## `AgentEval` Framework - -Our previous research on [assistive agents in Minecraft](https://github.com/microsoft/iglu-datasets) suggested that the most optimal way to obtain human judgments is to present humans with two agents side by side and ask for preferences. In this setup of pairwise comparison, humans can develop criteria to explain why they prefer the behavior of one agent over another. For instance, _'the first agent was faster in execution,'_ or _'the second agent moves more naturally.'_ So, the comparative nature led humans to come up with a list of criteria that helps to infer the utility of the task. With this idea in mind, we designed `AgentEval` (shown in Fig. 1), where we employ LLMs to help us understand, verify, and assess task *utility* for the multi-agent system. Namely: - -* The goal of `CriticAgent` is to suggest the list of criteria (Fig. 1), that can be used to assess task utility. This is an example of how `CriticAgent` is defined using `Autogen`: - -```python -critic = autogen.AssistantAgent( - name="critic", - llm_config={"config_list": config_list}, - system_message="""You are a helpful assistant. You suggest criteria for evaluating different tasks. They should be distinguishable, quantifiable, and not redundant. - Convert the evaluation criteria into a dictionary where the keys are the criteria. - The value of each key is a dictionary as follows {"description": criteria description, "accepted_values": possible accepted inputs for this key} - Make sure the keys are criteria for assessing the given task. "accepted_values" include the acceptable inputs for each key that are fine-grained and preferably multi-graded levels. "description" includes the criterion description. - Return only the dictionary.""" -) -``` - -Next, the critic is given successful and failed examples of the task execution; then, it is able to return a list of criteria (Fig. 1). For reference, use the [following notebook](https://github.com/microsoft/autogen/blob/main/notebook/agenteval_cq_math.ipynb). - -* The goal of `QuantifierAgent` is to quantify each of the suggested criteria (Fig. 1), providing us with an idea of the utility of this system for the given task. Here is an example of how it can be defined: - -```python -quantifier = autogen.AssistantAgent( - name="quantifier", - llm_config={"config_list": config_list}, - system_message = """You are a helpful assistant. You quantify the output of different tasks based on the given criteria. - The criterion is given in a dictionary format where each key is a distinct criteria. - The value of each key is a dictionary as follows {"description": criteria description , "accepted_values": possible accepted inputs for this key} - You are going to quantify each of the criteria for a given task based on the task description. - Return a dictionary where the keys are the criteria and the values are the assessed performance based on accepted values for each criteria. - Return only the dictionary.""" - -) -``` - -## `AgentEval` Results based on Math Problems Dataset - - As an example, after running CriticAgent, we obtained the following criteria to verify the results for math problem dataset: - -| Criteria | Description | Accepted Values| -|-----------|-----|----------------| -| Problem Interpretation | Ability to correctly interpret the problem | ["completely off", "slightly relevant", "relevant", "mostly accurate", "completely accurate"]| -| Mathematical Methodology | Adequacy of the chosen mathematical or algorithmic methodology for the question | ["inappropriate", "barely adequate", "adequate", "mostly effective", "completely effective"] | -| Calculation Correctness | Accuracy of calculations made and solutions given | ["completely incorrect", "mostly incorrect", "neither", "mostly correct", "completely correct"] | -| Explanation Clarity | Clarity and comprehensibility of explanations, including language use and structure | ["not at all clear", "slightly clear", "moderately clear", "very clear", "completely clear"] | -| Code Efficiency | Quality of code in terms of efficiency and elegance |["not at all efficient", "slightly efficient", "moderately efficient", "very efficient", "extremely efficient"] | -| Code Correctness | Correctness of the provided code | ["completely incorrect", "mostly incorrect", "partly correct", "mostly correct", "completely correct"] - - -Then, after running QuantifierAgent, we obtained the results presented in Fig. 3, where you can see three models: -* AgentChat -* ReAct -* GPT-4 Vanilla Solver - -Lighter colors represent estimates for failed cases, and brighter colors show how discovered criteria were quantified. - - -Fig.3 presents results based on overall math problems dataset `_s` stands for successful cases, `_f` - stands for failed cases
- -We note that while applying agentEval to math problems, the agent was not exposed to any ground truth information about the problem. As such, this figure illustrates an estimated performance of the three different agents, namely, Autogen (blue), Gpt-4 (red), and ReAct (green). We observe that by comparing the performance of any of the three agents in successful cases (dark bars of any color) versus unsuccessful cases (lighter version of the same bar), we note that AgentEval was able to assign higher quantification to successful cases than that of failed ones. This observation verifies AgentEval's ability for task utility prediction. Additionally, AgentEval allows us to go beyond just a binary definition of success, enabling a more in-depth comparison between successful and failed cases. - -It's important not only to identify what is not working but also to recognize what and why actually went well. - -## Limitations and Future Work -The current implementation of `AgentEval` has a number of limitations which are planning to overcome in the future: -* The list of criteria varies per run (unless you store a seed). We would recommend to run `CriticAgent` at least two times, and pick criteria you think is important for your domain. -* The results of the `QuantifierAgent` can vary with each run, so we recommend conducting multiple runs to observe the extent of result variations. - -To mitigate the limitations mentioned above, we are working on VerifierAgent, whose goal is to stabilize the results and provide additional explanations. - -## Summary -`CriticAgent` and `QuantifierAgent` can be applied to the logs of any type of application, providing you with an in-depth understanding of the utility your solution brings to the user for a given task. - -We would love to hear about how AgentEval works for your application. Any feedback would be useful for future development. Please contact us on our [Discord](http://aka.ms/autogen-dc). - - -## Previous Research - -``` -@InProceedings{pmlr-v176-kiseleva22a, - title = "Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021", - author = "Kiseleva, Julia and Li, Ziming and Aliannejadi, Mohammad and Mohanty, Shrestha and ter Hoeve, Maartje and Burtsev, Mikhail and Skrynnik, Alexey and Zholus, Artem and Panov, Aleksandr and Srinet, Kavya and Szlam, Arthur and Sun, Yuxuan and Hofmann, Katja and C{\^o}t{\'e}, Marc-Alexandre and Awadallah, Ahmed and Abdrazakov, Linar and Churin, Igor and Manggala, Putra and Naszadi, Kata and van der Meer, Michiel and Kim, Taewoon", - booktitle = "Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track", - pages = "146--161", - year = 2022, - editor = "Kiela, Douwe and Ciccone, Marco and Caputo, Barbara", - volume = 176, - series = "Proceedings of Machine Learning Research", - month = "06--14 Dec", - publisher = "PMLR", - pdf = {https://proceedings.mlr.press/v176/kiseleva22a/kiseleva22a.pdf}, - url = {https://proceedings.mlr.press/v176/kiseleva22a.html}. -} -``` - - -``` -@InProceedings{pmlr-v220-kiseleva22a, - title = "Interactive Grounded Language Understanding in a Collaborative Environment: Retrospective on Iglu 2022 Competition", - author = "Kiseleva, Julia and Skrynnik, Alexey and Zholus, Artem and Mohanty, Shrestha and Arabzadeh, Negar and C\^{o}t\'e, Marc-Alexandre and Aliannejadi, Mohammad and Teruel, Milagro and Li, Ziming and Burtsev, Mikhail and ter Hoeve, Maartje and Volovikova, Zoya and Panov, Aleksandr and Sun, Yuxuan and Srinet, Kavya and Szlam, Arthur and Awadallah, Ahmed and Rho, Seungeun and Kwon, Taehwan and Wontae Nam, Daniel and Bivort Haiek, Felipe and Zhang, Edwin and Abdrazakov, Linar and Qingyam, Guo and Zhang, Jason and Guo, Zhibin", - booktitle = "Proceedings of the NeurIPS 2022 Competitions Track", - pages = "204--216", - year = 2022, - editor = "Ciccone, Marco and Stolovitzky, Gustavo and Albrecht, Jacob", - volume = 220, - series = "Proceedings of Machine Learning Research", - month = "28 Nov--09 Dec", - publisher = "PMLR", - pdf = "https://proceedings.mlr.press/v220/kiseleva22a/kiseleva22a.pdf", - url = "https://proceedings.mlr.press/v220/kiseleva22a.html". -} -``` +--- +title: How to Assess Utility of LLM-powered Applications? +authors: + - julianakiseleva + - narabzad +tags: [LLM, GPT, evaluation, task utility] +--- + + + + +Fig.1 illustrates the general flow of AgentEval
+ +**TL;DR:** +* As a developer of an LLM-powered application, how can you assess the utility it brings to end users while helping them with their tasks? +* To shed light on the question above, we introduce `AgentEval` — the first version of the framework to assess the utility of any LLM-powered application crafted to assist users in specific tasks. AgentEval aims to simplify the evaluation process by automatically proposing a set of criteria tailored to the unique purpose of your application. This allows for a comprehensive assessment, quantifying the utility of your application against the suggested criteria. +* We demonstrate how `AgentEval` work using [math problems dataset](https://microsoft.github.io/autogen/blog/2023/06/28/MathChat) as an example in the [following notebook](https://github.com/microsoft/autogen/blob/main/notebook/agenteval_cq_math.ipynb). Any feedback would be useful for future development. Please contact us on our [Discord](http://aka.ms/autogen-dc). + + +## Introduction + + AutoGen aims to simplify the development of LLM-powered multi-agent systems for various applications, ultimately making end users' lives easier by assisting with their tasks. Next, we all yearn to understand how our developed systems perform, their utility for users, and, perhaps most crucially, how we can enhance them. Directly evaluating multi-agent systems poses challenges as current approaches predominantly rely on success metrics – essentially, whether the agent accomplishes tasks. However, comprehending user interaction with a system involves far more than success alone. Take math problems, for instance; it's not merely about the agent solving the problem. Equally significant is its ability to convey solutions based on various criteria, including completeness, conciseness, and the clarity of the provided explanation. Furthermore, success isn't always clearly defined for every task. + + Rapid advances in LLMs and multi-agent systems have brought forth many emerging capabilities that we're keen on translating into tangible utilities for end users. We introduce the first version of `AgentEval` framework - a tool crafted to empower developers in swiftly gauging the utility of LLM-powered applications designed to help end users accomplish the desired task. + + + +Fig. 2 provides an overview of the tasks taxonomy
+ + +Let's first look into an overview of the suggested task taxonomy that a multi-agent system can be designed for. In general, the tasks can be split into two types, where: +* _Success is not clearly defined_ - refer to instances when users utilize a system in an assistive manner, seeking suggestions rather than expecting the system to solve the task. For example, a user might request the system to generate an email. In many cases, this generated content serves as a template that the user will later edit. However, defining success precisely for such tasks is relatively complex. +* _Success is clearly defined_ - refer to instances where we can clearly define whether a system solved the task or not. Consider agents that assist in accomplishing household tasks, where the definition of success is clear and measurable. This category can be further divided into two separate subcategories: + * _The optimal solution exits_ - these are tasks where only one solution is possible. For example, if you ask your assistant to turn on the light, the success of this task is clearly defined, and there is only one way to accomplish it. + * _Multiple solutions exist_ - increasingly, we observe situations where multiple trajectories of agent behavior can lead to either success or failure. In such cases, it is crucial to differentiate between the various successful and unsuccessful trajectories. For example, when you ask the agent to suggest you a food recipe or tell you a joke. + +In our `AgentEval` framework, we are currently focusing on tasks where _Success is clearly defined_. Next, we will introduce the suggested framework. + +## `AgentEval` Framework + +Our previous research on [assistive agents in Minecraft](https://github.com/microsoft/iglu-datasets) suggested that the most optimal way to obtain human judgments is to present humans with two agents side by side and ask for preferences. In this setup of pairwise comparison, humans can develop criteria to explain why they prefer the behavior of one agent over another. For instance, _'the first agent was faster in execution,'_ or _'the second agent moves more naturally.'_ So, the comparative nature led humans to come up with a list of criteria that helps to infer the utility of the task. With this idea in mind, we designed `AgentEval` (shown in Fig. 1), where we employ LLMs to help us understand, verify, and assess task *utility* for the multi-agent system. Namely: + +* The goal of `CriticAgent` is to suggest the list of criteria (Fig. 1), that can be used to assess task utility. This is an example of how `CriticAgent` is defined using `Autogen`: + +```python +critic = autogen.AssistantAgent( + name="critic", + llm_config={"config_list": config_list}, + system_message="""You are a helpful assistant. You suggest criteria for evaluating different tasks. They should be distinguishable, quantifiable, and not redundant. + Convert the evaluation criteria into a dictionary where the keys are the criteria. + The value of each key is a dictionary as follows {"description": criteria description, "accepted_values": possible accepted inputs for this key} + Make sure the keys are criteria for assessing the given task. "accepted_values" include the acceptable inputs for each key that are fine-grained and preferably multi-graded levels. "description" includes the criterion description. + Return only the dictionary.""" +) +``` + +Next, the critic is given successful and failed examples of the task execution; then, it is able to return a list of criteria (Fig. 1). For reference, use the [following notebook](https://github.com/microsoft/autogen/blob/main/notebook/agenteval_cq_math.ipynb). + +* The goal of `QuantifierAgent` is to quantify each of the suggested criteria (Fig. 1), providing us with an idea of the utility of this system for the given task. Here is an example of how it can be defined: + +```python +quantifier = autogen.AssistantAgent( + name="quantifier", + llm_config={"config_list": config_list}, + system_message = """You are a helpful assistant. You quantify the output of different tasks based on the given criteria. + The criterion is given in a dictionary format where each key is a distinct criteria. + The value of each key is a dictionary as follows {"description": criteria description , "accepted_values": possible accepted inputs for this key} + You are going to quantify each of the criteria for a given task based on the task description. + Return a dictionary where the keys are the criteria and the values are the assessed performance based on accepted values for each criteria. + Return only the dictionary.""" + +) +``` + +## `AgentEval` Results based on Math Problems Dataset + + As an example, after running CriticAgent, we obtained the following criteria to verify the results for math problem dataset: + +| Criteria | Description | Accepted Values| +|-----------|-----|----------------| +| Problem Interpretation | Ability to correctly interpret the problem | ["completely off", "slightly relevant", "relevant", "mostly accurate", "completely accurate"]| +| Mathematical Methodology | Adequacy of the chosen mathematical or algorithmic methodology for the question | ["inappropriate", "barely adequate", "adequate", "mostly effective", "completely effective"] | +| Calculation Correctness | Accuracy of calculations made and solutions given | ["completely incorrect", "mostly incorrect", "neither", "mostly correct", "completely correct"] | +| Explanation Clarity | Clarity and comprehensibility of explanations, including language use and structure | ["not at all clear", "slightly clear", "moderately clear", "very clear", "completely clear"] | +| Code Efficiency | Quality of code in terms of efficiency and elegance |["not at all efficient", "slightly efficient", "moderately efficient", "very efficient", "extremely efficient"] | +| Code Correctness | Correctness of the provided code | ["completely incorrect", "mostly incorrect", "partly correct", "mostly correct", "completely correct"] + + +Then, after running QuantifierAgent, we obtained the results presented in Fig. 3, where you can see three models: +* AgentChat +* ReAct +* GPT-4 Vanilla Solver + +Lighter colors represent estimates for failed cases, and brighter colors show how discovered criteria were quantified. + + +Fig.3 presents results based on overall math problems dataset `_s` stands for successful cases, `_f` - stands for failed cases
+ +We note that while applying agentEval to math problems, the agent was not exposed to any ground truth information about the problem. As such, this figure illustrates an estimated performance of the three different agents, namely, Autogen (blue), Gpt-4 (red), and ReAct (green). We observe that by comparing the performance of any of the three agents in successful cases (dark bars of any color) versus unsuccessful cases (lighter version of the same bar), we note that AgentEval was able to assign higher quantification to successful cases than that of failed ones. This observation verifies AgentEval's ability for task utility prediction. Additionally, AgentEval allows us to go beyond just a binary definition of success, enabling a more in-depth comparison between successful and failed cases. + +It's important not only to identify what is not working but also to recognize what and why actually went well. + +## Limitations and Future Work +The current implementation of `AgentEval` has a number of limitations which are planning to overcome in the future: +* The list of criteria varies per run (unless you store a seed). We would recommend to run `CriticAgent` at least two times, and pick criteria you think is important for your domain. +* The results of the `QuantifierAgent` can vary with each run, so we recommend conducting multiple runs to observe the extent of result variations. + +To mitigate the limitations mentioned above, we are working on VerifierAgent, whose goal is to stabilize the results and provide additional explanations. + +## Summary +`CriticAgent` and `QuantifierAgent` can be applied to the logs of any type of application, providing you with an in-depth understanding of the utility your solution brings to the user for a given task. + +We would love to hear about how AgentEval works for your application. Any feedback would be useful for future development. Please contact us on our [Discord](http://aka.ms/autogen-dc). + + +## Previous Research + +``` +@InProceedings{pmlr-v176-kiseleva22a, + title = "Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021", + author = "Kiseleva, Julia and Li, Ziming and Aliannejadi, Mohammad and Mohanty, Shrestha and ter Hoeve, Maartje and Burtsev, Mikhail and Skrynnik, Alexey and Zholus, Artem and Panov, Aleksandr and Srinet, Kavya and Szlam, Arthur and Sun, Yuxuan and Hofmann, Katja and C{\^o}t{\'e}, Marc-Alexandre and Awadallah, Ahmed and Abdrazakov, Linar and Churin, Igor and Manggala, Putra and Naszadi, Kata and van der Meer, Michiel and Kim, Taewoon", + booktitle = "Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track", + pages = "146--161", + year = 2022, + editor = "Kiela, Douwe and Ciccone, Marco and Caputo, Barbara", + volume = 176, + series = "Proceedings of Machine Learning Research", + month = "06--14 Dec", + publisher = "PMLR", + pdf = {https://proceedings.mlr.press/v176/kiseleva22a/kiseleva22a.pdf}, + url = {https://proceedings.mlr.press/v176/kiseleva22a.html}. +} +``` + + +``` +@InProceedings{pmlr-v220-kiseleva22a, + title = "Interactive Grounded Language Understanding in a Collaborative Environment: Retrospective on Iglu 2022 Competition", + author = "Kiseleva, Julia and Skrynnik, Alexey and Zholus, Artem and Mohanty, Shrestha and Arabzadeh, Negar and C\^{o}t\'e, Marc-Alexandre and Aliannejadi, Mohammad and Teruel, Milagro and Li, Ziming and Burtsev, Mikhail and ter Hoeve, Maartje and Volovikova, Zoya and Panov, Aleksandr and Sun, Yuxuan and Srinet, Kavya and Szlam, Arthur and Awadallah, Ahmed and Rho, Seungeun and Kwon, Taehwan and Wontae Nam, Daniel and Bivort Haiek, Felipe and Zhang, Edwin and Abdrazakov, Linar and Qingyam, Guo and Zhang, Jason and Guo, Zhibin", + booktitle = "Proceedings of the NeurIPS 2022 Competitions Track", + pages = "204--216", + year = 2022, + editor = "Ciccone, Marco and Stolovitzky, Gustavo and Albrecht, Jacob", + volume = 220, + series = "Proceedings of Machine Learning Research", + month = "28 Nov--09 Dec", + publisher = "PMLR", + pdf = "https://proceedings.mlr.press/v220/kiseleva22a/kiseleva22a.pdf", + url = "https://proceedings.mlr.press/v220/kiseleva22a.html". +} +``` diff --git a/website/blog/2023-12-01-AutoGenStudio/index.mdx b/website/blog/2023-12-01-AutoGenStudio/index.mdx index 4d893b144a78..a2558acb01d6 100644 --- a/website/blog/2023-12-01-AutoGenStudio/index.mdx +++ b/website/blog/2023-12-01-AutoGenStudio/index.mdx @@ -1,237 +1,237 @@ ---- -title: "AutoGen Studio: Interactively Explore Multi-Agent Workflows" -authors: - - victordibia - - gagb - - samershi -tags: [AutoGen, UI, web, UX] ---- - - - -- - AutoGen Studio: Solving a task with multiple agents that generate a pdf - document with images. - -
- -## TL;DR - -To help you rapidly prototype multi-agent solutions for your tasks, we are introducing AutoGen Studio, an interface powered by [AutoGen](https://github.com/microsoft/autogen/tree/main/autogen). It allows you to: - -- Declaratively define and modify agents and multi-agent workflows through a point and click, drag and drop interface (e.g., you can select the parameters of two agents that will communicate to solve your task). -- Use our UI to create chat sessions with the specified agents and view results (e.g., view chat history, generated files, and time taken). -- Explicitly add skills to your agents and accomplish more tasks. -- Publish your sessions to a local gallery. - - -See the official AutoGen Studio documentation [here](https://microsoft.github.io/autogen/docs/autogen-studio/getting-started) for more details. - -AutoGen Studio is open source [code here](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio), and can be installed via pip. Give it a try! - -```bash -pip install autogenstudio -``` - -## Introduction - -The accelerating pace of technology has ushered us into an era where digital assistants (or agents) are becoming integral to our lives. [AutoGen](https://github.com/microsoft/autogen/tree/main/autogen) has emerged as a leading framework for orchestrating the power of agents. In the spirit of expanding this frontier and democratizing this capability, we are thrilled to introduce a new user-friendly interface: **AutoGen Studio**. - -With AutoGen Studio, users can rapidly create, manage, and interact with agents that can learn, adapt, and collaborate. As we release this interface into the open-source community, our ambition is not only to enhance productivity but to inspire a level of personalized interaction between humans and agents. - -> **Note**: AutoGen Studio is meant to help you rapidly prototype multi-agent workflows and demonstrate an example of end user interfaces built with AutoGen. It is not meant to be a production-ready app. - -## Getting Started with AutoGen Studio - -The following guide will help you get AutoGen Studio up and running on your system. - -### Configuring an LLM Provider - -To get started, you need access to a language model. You can get this set up by following the steps in the AutoGen documentation [here](https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints). Configure your environment with either `OPENAI_API_KEY` or `AZURE_OPENAI_API_KEY`. - -For example, in your terminal, you would set the API key like this: - -```bash -export OPENAI_API_KEY=- - AutoGen Studio Build View: View, add or edit skills that an agent can - leverage in addressing tasks. - -
- -**Agents**: This provides an interface to declaratively specify properties for an AutoGen agent (mirrors most of the members of a base [AutoGen conversable agent](https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py) class). - -**Agent Workflows**: An agent workflow is a specification of a set of agents that can work together to accomplish a task. The simplest version of this is a setup with two agents – a user proxy agent (that represents a user i.e. it compiles code and prints result) and an assistant that can address task requests (e.g., generating plans, writing code, evaluating responses, proposing error recovery steps, etc.). A more complex flow could be a group chat where even more agents work towards a solution. - -### Playground - - - -- - AutoGen Studio Playground View: Agents collaborate, use available skills - (ability to generate images) to address a user task (generate pdf's). - -
- -The playground section is focused on interacting with agent workflows defined in the previous build section. It includes the following concepts: - -**Session**: A session refers to a period of continuous interaction or engagement with an agent workflow, typically characterized by a sequence of activities or operations aimed at achieving specific objectives. It includes the agent workflow configuration, the interactions between the user and the agents. A session can be “published” to a “gallery”. - -**Chat View**: A chat is a sequence of interactions between a user and an agent. It is a part of a session. - -### Gallery - -This section is focused on sharing and reusing artifacts (e.g., workflow configurations, sessions, etc.). - -AutoGen Studio comes with 3 example skills: `fetch_profile`, `find_papers`, `generate_images`. Please feel free to review the repo to learn more about how they work. - -## The AutoGen Studio API - -While AutoGen Studio is a web interface, it is powered by an underlying python API that is reusable and modular. Importantly, we have implemented an API where agent workflows can be declaratively specified (in JSON), loaded and run. An example of the current API is shown below. Please consult the [AutoGen Studio repo](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio) for more details. - -```python -import json -from autogenstudio import AutoGenWorkFlowManager, AgentWorkFlowConfig - -# load an agent specification in JSON -agent_spec = json.load(open('agent_spec.json')) - -# Create an AutoGen Workflow Configuration from the agent specification -agent_work_flow_config = FlowConfig(**agent_spec) - -# Create a Workflow from the configuration -agent_work_flow = AutoGenWorkFlowManager(agent_work_flow_config) - -# Run the workflow on a task -task_query = "What is the height of the Eiffel Tower?" -agent_work_flow.run(message=task_query) -``` - -## Road Map and Next Steps - -As we continue to develop and refine AutoGen Studio, the road map below outlines an array of enhancements and new features planned for future releases. Here's what users can look forward to: - -- **Complex Agent Workflows**: We're working on integrating support for more sophisticated agent workflows, such as `GroupChat`, allowing for richer interaction between multiple agents or dynamic topologies. -- **Improved User Experience**: This includes features like streaming intermediate model output for real-time feedback, better summarization of agent responses, information on costs of each interaction. We will also invest in improving the workflow for composing and reusing agents. We will also explore support for more interactive human in the loop feedback to agents. -- **Expansion of Agent Skills**: We will work towards improving the workflow for authoring, composing and reusing agent skills. -- **Community Features**: Facilitation of sharing and collaboration within AutoGen Studio user community is a key goal. We're exploring options for sharing sessions and results more easily among users and contributing to a shared repository of skills, agents, and agent workflows. - -## Contribution Guide - -We welcome contributions to AutoGen Studio. We recommend the following general steps to contribute to the project: - -- Review the overall AutoGen project [contribution guide](https://github.com/microsoft/autogen?tab=readme-ov-file#contributing). -- Please review the AutoGen Studio [roadmap](https://github.com/microsoft/autogen/issues/737) to get a sense of the current priorities for the project. Help is appreciated especially with Studio issues tagged with `help-wanted`. -- Please initiate a discussion on the roadmap issue or a new issue to discuss your proposed contribution. -- Please review the autogenstudio dev branch here [dev branch].(https://github.com/microsoft/autogen/tree/autogenstudio) and use as a base for your contribution. This way, your contribution will be aligned with the latest changes in the AutoGen Studio project. -- Submit a pull request with your contribution! -- If you are modifying AutoGen Studio in vscode, it has its own devcontainer to simplify dev work. See instructions in `.devcontainer/README.md` on how to use it. -- Please use the tag `studio` for any issues, questions, and PRs related to Studio. - -### FAQ - -**Q: Where can I adjust the default skills, agent and workflow configurations?** -A: You can modify agent configurations directly from the UI or by editing the `autogentstudio/utils/dbdefaults.json` file which is used to initialize the database. - -**Q: If I want to reset the entire conversation with an agent, how do I go about it?** -A: To reset your conversation history, you can delete the `database.sqlite` file. If you need to clear user-specific data, remove the relevant `autogenstudio/web/files/user/+ + AutoGen Studio: Solving a task with multiple agents that generate a pdf + document with images. + +
+ +## TL;DR + +To help you rapidly prototype multi-agent solutions for your tasks, we are introducing AutoGen Studio, an interface powered by [AutoGen](https://github.com/microsoft/autogen/tree/main/autogen). It allows you to: + +- Declaratively define and modify agents and multi-agent workflows through a point and click, drag and drop interface (e.g., you can select the parameters of two agents that will communicate to solve your task). +- Use our UI to create chat sessions with the specified agents and view results (e.g., view chat history, generated files, and time taken). +- Explicitly add skills to your agents and accomplish more tasks. +- Publish your sessions to a local gallery. + + +See the official AutoGen Studio documentation [here](https://microsoft.github.io/autogen/docs/autogen-studio/getting-started) for more details. + +AutoGen Studio is open source [code here](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio), and can be installed via pip. Give it a try! + +```bash +pip install autogenstudio +``` + +## Introduction + +The accelerating pace of technology has ushered us into an era where digital assistants (or agents) are becoming integral to our lives. [AutoGen](https://github.com/microsoft/autogen/tree/main/autogen) has emerged as a leading framework for orchestrating the power of agents. In the spirit of expanding this frontier and democratizing this capability, we are thrilled to introduce a new user-friendly interface: **AutoGen Studio**. + +With AutoGen Studio, users can rapidly create, manage, and interact with agents that can learn, adapt, and collaborate. As we release this interface into the open-source community, our ambition is not only to enhance productivity but to inspire a level of personalized interaction between humans and agents. + +> **Note**: AutoGen Studio is meant to help you rapidly prototype multi-agent workflows and demonstrate an example of end user interfaces built with AutoGen. It is not meant to be a production-ready app. + +## Getting Started with AutoGen Studio + +The following guide will help you get AutoGen Studio up and running on your system. + +### Configuring an LLM Provider + +To get started, you need access to a language model. You can get this set up by following the steps in the AutoGen documentation [here](https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints). Configure your environment with either `OPENAI_API_KEY` or `AZURE_OPENAI_API_KEY`. + +For example, in your terminal, you would set the API key like this: + +```bash +export OPENAI_API_KEY=+ + AutoGen Studio Build View: View, add or edit skills that an agent can + leverage in addressing tasks. + +
+ +**Agents**: This provides an interface to declaratively specify properties for an AutoGen agent (mirrors most of the members of a base [AutoGen conversable agent](https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py) class). + +**Agent Workflows**: An agent workflow is a specification of a set of agents that can work together to accomplish a task. The simplest version of this is a setup with two agents – a user proxy agent (that represents a user i.e. it compiles code and prints result) and an assistant that can address task requests (e.g., generating plans, writing code, evaluating responses, proposing error recovery steps, etc.). A more complex flow could be a group chat where even more agents work towards a solution. + +### Playground + + + ++ + AutoGen Studio Playground View: Agents collaborate, use available skills + (ability to generate images) to address a user task (generate pdf's). + +
+ +The playground section is focused on interacting with agent workflows defined in the previous build section. It includes the following concepts: + +**Session**: A session refers to a period of continuous interaction or engagement with an agent workflow, typically characterized by a sequence of activities or operations aimed at achieving specific objectives. It includes the agent workflow configuration, the interactions between the user and the agents. A session can be “published” to a “gallery”. + +**Chat View**: A chat is a sequence of interactions between a user and an agent. It is a part of a session. + +### Gallery + +This section is focused on sharing and reusing artifacts (e.g., workflow configurations, sessions, etc.). + +AutoGen Studio comes with 3 example skills: `fetch_profile`, `find_papers`, `generate_images`. Please feel free to review the repo to learn more about how they work. + +## The AutoGen Studio API + +While AutoGen Studio is a web interface, it is powered by an underlying python API that is reusable and modular. Importantly, we have implemented an API where agent workflows can be declaratively specified (in JSON), loaded and run. An example of the current API is shown below. Please consult the [AutoGen Studio repo](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio) for more details. + +```python +import json +from autogenstudio import AutoGenWorkFlowManager, AgentWorkFlowConfig + +# load an agent specification in JSON +agent_spec = json.load(open('agent_spec.json')) + +# Create an AutoGen Workflow Configuration from the agent specification +agent_work_flow_config = FlowConfig(**agent_spec) + +# Create a Workflow from the configuration +agent_work_flow = AutoGenWorkFlowManager(agent_work_flow_config) + +# Run the workflow on a task +task_query = "What is the height of the Eiffel Tower?" +agent_work_flow.run(message=task_query) +``` + +## Road Map and Next Steps + +As we continue to develop and refine AutoGen Studio, the road map below outlines an array of enhancements and new features planned for future releases. Here's what users can look forward to: + +- **Complex Agent Workflows**: We're working on integrating support for more sophisticated agent workflows, such as `GroupChat`, allowing for richer interaction between multiple agents or dynamic topologies. +- **Improved User Experience**: This includes features like streaming intermediate model output for real-time feedback, better summarization of agent responses, information on costs of each interaction. We will also invest in improving the workflow for composing and reusing agents. We will also explore support for more interactive human in the loop feedback to agents. +- **Expansion of Agent Skills**: We will work towards improving the workflow for authoring, composing and reusing agent skills. +- **Community Features**: Facilitation of sharing and collaboration within AutoGen Studio user community is a key goal. We're exploring options for sharing sessions and results more easily among users and contributing to a shared repository of skills, agents, and agent workflows. + +## Contribution Guide + +We welcome contributions to AutoGen Studio. We recommend the following general steps to contribute to the project: + +- Review the overall AutoGen project [contribution guide](https://github.com/microsoft/autogen?tab=readme-ov-file#contributing). +- Please review the AutoGen Studio [roadmap](https://github.com/microsoft/autogen/issues/737) to get a sense of the current priorities for the project. Help is appreciated especially with Studio issues tagged with `help-wanted`. +- Please initiate a discussion on the roadmap issue or a new issue to discuss your proposed contribution. +- Please review the autogenstudio dev branch here [dev branch].(https://github.com/microsoft/autogen/tree/autogenstudio) and use as a base for your contribution. This way, your contribution will be aligned with the latest changes in the AutoGen Studio project. +- Submit a pull request with your contribution! +- If you are modifying AutoGen Studio in vscode, it has its own devcontainer to simplify dev work. See instructions in `.devcontainer/README.md` on how to use it. +- Please use the tag `studio` for any issues, questions, and PRs related to Studio. + +### FAQ + +**Q: Where can I adjust the default skills, agent and workflow configurations?** +A: You can modify agent configurations directly from the UI or by editing the `autogentstudio/utils/dbdefaults.json` file which is used to initialize the database. + +**Q: If I want to reset the entire conversation with an agent, how do I go about it?** +A: To reset your conversation history, you can delete the `database.sqlite` file. If you need to clear user-specific data, remove the relevant `autogenstudio/web/files/user/- - AutoGenBench is a standalone tool for evaluating AutoGen agents and - workflows on common benchmarks. - -
- -## TL;DR - -Today we are releasing AutoGenBench - a tool for evaluating AutoGen agents and workflows on established LLM and agentic benchmarks. - -AutoGenBench is a standalone command line tool, installable from PyPI, which handles downloading, configuring, running, and reporting supported benchmarks. AutoGenBench works best when run alongside Docker, since it uses Docker to isolate tests from one another. - -- See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for information on installation and running benchmarks. -- See the [AutoGenBench CONTRIBUTING guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) for information on developing or contributing benchmark datasets. - -### Quick Start - -Get started quickly by running the following commands in a bash terminal. - -_Note:_ You may need to adjust the path to the `OAI_CONFIG_LIST`, as appropriate. - -```sh -export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST) -pip install autogenbench -autogenbench clone HumanEval -cd HumanEval -cat README.md -autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl -autogenbench tabulate Results/human_eval_two_agents -``` - -## Introduction - -Measurement and evaluation are core components of every major AI or ML research project. The same is true for AutoGen. To this end, today we are releasing AutoGenBench, a standalone command line tool that we have been using to guide development of AutoGen. Conveniently, AutoGenBench handles: downloading, configuring, running, and reporting results of agents on various public benchmark datasets. In addition to reporting top-line numbers, each AutoGenBench run produces a comprehensive set of logs and telemetry that can be used for debugging, profiling, computing custom metrics, and as input to [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval). In the remainder of this blog post, we outline core design principles for AutoGenBench (key to understanding its operation); present a guide to installing and running AutoGenBench; outline a roadmap for evaluation; and conclude with an open call for contributions. - -## Design Principles - -AutoGenBench is designed around three core design principles. Knowing these principles will help you understand the tool, its operation and its output. These three principles are: - -- **Repetition:** LLMs are stochastic, and in many cases, so too is the code they write to solve problems. For example, a Python script might call an external search engine, and the results may vary run-to-run. This can lead to variance in agent performance. Repetition is key to measuring and understanding this variance. To this end, AutoGenBench is built from the ground up with an understanding that tasks may be run multiple times, and that variance is a metric we often want to measure. - -- **Isolation:** Agents interact with their worlds in both subtle and overt ways. For example an agent may install a python library or write a file to disk. This can lead to ordering effects that can impact future measurements. Consider, for example, comparing two agents on a common benchmark. One agent may appear more efficient than the other simply because it ran second, and benefitted from the hard work the first agent did in installing and debugging necessary Python libraries. To address this, AutoGenBench isolates each task in its own Docker container. This ensures that all runs start with the same initial conditions. (Docker is also a _much safer way to run agent-produced code_, in general.) - -- **Instrumentation:** While top-line metrics are great for comparing agents or models, we often want much more information about how the agents are performing, where they are getting stuck, and how they can be improved. We may also later think of new research questions that require computing a different set of metrics. To this end, AutoGenBench is designed to log everything, and to compute metrics from those logs. This ensures that one can always go back to the logs to answer questions about what happened, run profiling software, or feed the logs into tools like [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval). - -## Installing and Running AutoGenBench - -As noted above, isolation is a key design principle, and so AutoGenBench must be run in an environment where Docker is available (desktop or Engine). **It will not run in GitHub codespaces**, unless you opt for native execution (which is strongly discouraged). To install Docker Desktop see [https://www.docker.com/products/docker-desktop/](https://www.docker.com/products/docker-desktop/). -Once Docker is installed, AutoGenBench can then be installed as a standalone tool from PyPI. With `pip`, installation can be achieved as follows: - -```sh -pip install autogenbench -``` - -After installation, you must configure your API keys. As with other AutoGen applications, AutoGenBench will look for the OpenAI keys in the OAI_CONFIG_LIST file in the current working directory, or the OAI_CONFIG_LIST environment variable. This behavior can be overridden using a command-line parameter. - -If you will be running multiple benchmarks, it is often most convenient to leverage the environment variable option. You can load your keys into the environment variable by executing: - -```sh -export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST) -``` - -## A Typical Session - -Once AutoGenBench and necessary keys are installed, a typical session will look as follows: - -``` -autogenbench clone HumanEval -cd HumanEval -cat README.md -autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl -autogenbench tabulate results/human_eval_two_agents -``` - -Where: - -- `autogenbench clone HumanEval` downloads and expands the HumanEval benchmark scenario. -- `cd HumanEval; cat README.md` navigates to the benchmark directory, and prints the README (which you should always read!) -- `autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl` - runs a 10% subsample of the tasks defined in `Tasks/human_eval_two_agents.jsonl`. Each task is run 3 times. -- `autogenbench tabulate results/human_eval_two_agents` tabulates the results of the run. - -After running the above `tabulate` command, you should see output similar to the following: - -``` - Trial 0 Trial 1 Trial 2 -Task Id Success Success Success -------------- --------- --------- --------- -HumanEval_107 False True True -HumanEval_22 True True True -HumanEval_43 True True True -HumanEval_88 True True True -HumanEval_14 True True True -HumanEval_157 True True True -HumanEval_141 True True True -HumanEval_57 True True True -HumanEval_154 True True True -HumanEval_153 True True True -HumanEval_93 False True False -HumanEval_137 True True True -HumanEval_143 True True True -HumanEval_13 True True True -HumanEval_49 True True True -HumanEval_95 True True True -------------- --------- --------- --------- -Successes 14 16 15 -Failures 2 0 1 -Missing 0 0 0 -Total 16 16 16 - -CAUTION: 'autogenbench tabulate' is in early preview. -Please do not cite these values in academic work without first inspecting and verifying the results in the logs yourself. -``` - -From this output we can see the results of the three separate repetitions of each task, and final summary statistics of each run. In this case, the results were generated via GPT-4 (as defined in the OAI_CONFIG_LIST that was provided), and used the `TwoAgents` template. **It is important to remember that AutoGenBench evaluates _specific_ end-to-end configurations of agents (as opposed to evaluating a model or cognitive framework more generally).** - -Finally, complete execution traces and logs can be found in the `Results` folder. See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for more details about command-line options and output formats. Each of these commands also offers extensive in-line help via: - -- `autogenbench --help` -- `autogenbench clone --help` -- `autogenbench run --help` -- `autogenbench tabulate --help` - -## Roadmap - -While we are announcing AutoGenBench, we note that it is very much an evolving project in its own right. Over the next few weeks and months we hope to: - -- Onboard many additional benchmarks beyond those shipping today -- Greatly improve logging and telemetry -- Introduce new core metrics including total costs, task completion time, conversation turns, etc. -- Provide tighter integration with AgentEval and AutoGen Studio - -For an up to date tracking of our work items on this project, please see [AutoGenBench Work Items](https://github.com/microsoft/autogen/issues/973) - -## Call for Participation - -Finally, we want to end this blog post with an open call for contributions. AutoGenBench is still nascent, and has much opportunity for improvement. New benchmarks are constantly being published, and will need to be added. Everyone may have their own distinct set of metrics that they care most about optimizing, and these metrics should be onboarded. To this end, we welcome any and all contributions to this corner of the AutoGen project. If contributing is something that interests you, please see the [contributor’s guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) and join our [Discord](https://aka.ms/autogen-dc) discussion in the [#autogenbench](https://discord.com/channels/1153072414184452236/1199851779328847902) channel! +--- +title: "AutoGenBench -- A Tool for Measuring and Evaluating AutoGen Agents" +authors: + - afourney + - qingyunwu +tags: [AutoGen] +--- + + + ++ + AutoGenBench is a standalone tool for evaluating AutoGen agents and + workflows on common benchmarks. + +
+ +## TL;DR + +Today we are releasing AutoGenBench - a tool for evaluating AutoGen agents and workflows on established LLM and agentic benchmarks. + +AutoGenBench is a standalone command line tool, installable from PyPI, which handles downloading, configuring, running, and reporting supported benchmarks. AutoGenBench works best when run alongside Docker, since it uses Docker to isolate tests from one another. + +- See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for information on installation and running benchmarks. +- See the [AutoGenBench CONTRIBUTING guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) for information on developing or contributing benchmark datasets. + +### Quick Start + +Get started quickly by running the following commands in a bash terminal. + +_Note:_ You may need to adjust the path to the `OAI_CONFIG_LIST`, as appropriate. + +```sh +export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST) +pip install autogenbench +autogenbench clone HumanEval +cd HumanEval +cat README.md +autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl +autogenbench tabulate Results/human_eval_two_agents +``` + +## Introduction + +Measurement and evaluation are core components of every major AI or ML research project. The same is true for AutoGen. To this end, today we are releasing AutoGenBench, a standalone command line tool that we have been using to guide development of AutoGen. Conveniently, AutoGenBench handles: downloading, configuring, running, and reporting results of agents on various public benchmark datasets. In addition to reporting top-line numbers, each AutoGenBench run produces a comprehensive set of logs and telemetry that can be used for debugging, profiling, computing custom metrics, and as input to [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval). In the remainder of this blog post, we outline core design principles for AutoGenBench (key to understanding its operation); present a guide to installing and running AutoGenBench; outline a roadmap for evaluation; and conclude with an open call for contributions. + +## Design Principles + +AutoGenBench is designed around three core design principles. Knowing these principles will help you understand the tool, its operation and its output. These three principles are: + +- **Repetition:** LLMs are stochastic, and in many cases, so too is the code they write to solve problems. For example, a Python script might call an external search engine, and the results may vary run-to-run. This can lead to variance in agent performance. Repetition is key to measuring and understanding this variance. To this end, AutoGenBench is built from the ground up with an understanding that tasks may be run multiple times, and that variance is a metric we often want to measure. + +- **Isolation:** Agents interact with their worlds in both subtle and overt ways. For example an agent may install a python library or write a file to disk. This can lead to ordering effects that can impact future measurements. Consider, for example, comparing two agents on a common benchmark. One agent may appear more efficient than the other simply because it ran second, and benefitted from the hard work the first agent did in installing and debugging necessary Python libraries. To address this, AutoGenBench isolates each task in its own Docker container. This ensures that all runs start with the same initial conditions. (Docker is also a _much safer way to run agent-produced code_, in general.) + +- **Instrumentation:** While top-line metrics are great for comparing agents or models, we often want much more information about how the agents are performing, where they are getting stuck, and how they can be improved. We may also later think of new research questions that require computing a different set of metrics. To this end, AutoGenBench is designed to log everything, and to compute metrics from those logs. This ensures that one can always go back to the logs to answer questions about what happened, run profiling software, or feed the logs into tools like [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval). + +## Installing and Running AutoGenBench + +As noted above, isolation is a key design principle, and so AutoGenBench must be run in an environment where Docker is available (desktop or Engine). **It will not run in GitHub codespaces**, unless you opt for native execution (which is strongly discouraged). To install Docker Desktop see [https://www.docker.com/products/docker-desktop/](https://www.docker.com/products/docker-desktop/). +Once Docker is installed, AutoGenBench can then be installed as a standalone tool from PyPI. With `pip`, installation can be achieved as follows: + +```sh +pip install autogenbench +``` + +After installation, you must configure your API keys. As with other AutoGen applications, AutoGenBench will look for the OpenAI keys in the OAI_CONFIG_LIST file in the current working directory, or the OAI_CONFIG_LIST environment variable. This behavior can be overridden using a command-line parameter. + +If you will be running multiple benchmarks, it is often most convenient to leverage the environment variable option. You can load your keys into the environment variable by executing: + +```sh +export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST) +``` + +## A Typical Session + +Once AutoGenBench and necessary keys are installed, a typical session will look as follows: + +``` +autogenbench clone HumanEval +cd HumanEval +cat README.md +autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl +autogenbench tabulate results/human_eval_two_agents +``` + +Where: + +- `autogenbench clone HumanEval` downloads and expands the HumanEval benchmark scenario. +- `cd HumanEval; cat README.md` navigates to the benchmark directory, and prints the README (which you should always read!) +- `autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl` + runs a 10% subsample of the tasks defined in `Tasks/human_eval_two_agents.jsonl`. Each task is run 3 times. +- `autogenbench tabulate results/human_eval_two_agents` tabulates the results of the run. + +After running the above `tabulate` command, you should see output similar to the following: + +``` + Trial 0 Trial 1 Trial 2 +Task Id Success Success Success +------------- --------- --------- --------- +HumanEval_107 False True True +HumanEval_22 True True True +HumanEval_43 True True True +HumanEval_88 True True True +HumanEval_14 True True True +HumanEval_157 True True True +HumanEval_141 True True True +HumanEval_57 True True True +HumanEval_154 True True True +HumanEval_153 True True True +HumanEval_93 False True False +HumanEval_137 True True True +HumanEval_143 True True True +HumanEval_13 True True True +HumanEval_49 True True True +HumanEval_95 True True True +------------- --------- --------- --------- +Successes 14 16 15 +Failures 2 0 1 +Missing 0 0 0 +Total 16 16 16 + +CAUTION: 'autogenbench tabulate' is in early preview. +Please do not cite these values in academic work without first inspecting and verifying the results in the logs yourself. +``` + +From this output we can see the results of the three separate repetitions of each task, and final summary statistics of each run. In this case, the results were generated via GPT-4 (as defined in the OAI_CONFIG_LIST that was provided), and used the `TwoAgents` template. **It is important to remember that AutoGenBench evaluates _specific_ end-to-end configurations of agents (as opposed to evaluating a model or cognitive framework more generally).** + +Finally, complete execution traces and logs can be found in the `Results` folder. See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for more details about command-line options and output formats. Each of these commands also offers extensive in-line help via: + +- `autogenbench --help` +- `autogenbench clone --help` +- `autogenbench run --help` +- `autogenbench tabulate --help` + +## Roadmap + +While we are announcing AutoGenBench, we note that it is very much an evolving project in its own right. Over the next few weeks and months we hope to: + +- Onboard many additional benchmarks beyond those shipping today +- Greatly improve logging and telemetry +- Introduce new core metrics including total costs, task completion time, conversation turns, etc. +- Provide tighter integration with AgentEval and AutoGen Studio + +For an up to date tracking of our work items on this project, please see [AutoGenBench Work Items](https://github.com/microsoft/autogen/issues/973) + +## Call for Participation + +Finally, we want to end this blog post with an open call for contributions. AutoGenBench is still nascent, and has much opportunity for improvement. New benchmarks are constantly being published, and will need to be added. Everyone may have their own distinct set of metrics that they care most about optimizing, and these metrics should be onboarded. To this end, we welcome any and all contributions to this corner of the AutoGen project. If contributing is something that interests you, please see the [contributor’s guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) and join our [Discord](https://aka.ms/autogen-dc) discussion in the [#autogenbench](https://discord.com/channels/1153072414184452236/1199851779328847902) channel! diff --git a/website/blog/2024-02-11-FSM-GroupChat/index.mdx b/website/blog/2024-02-11-FSM-GroupChat/index.mdx index 74b5b49e35f5..3f2f7a5dba1e 100644 --- a/website/blog/2024-02-11-FSM-GroupChat/index.mdx +++ b/website/blog/2024-02-11-FSM-GroupChat/index.mdx @@ -1,288 +1,288 @@ ---- -title: "FSM Group Chat -- User-specified agent transitions" -authors: - - joshkyh - - freedeaths -tags: [AutoGen] ---- - - -Finite State Machine (FSM) Group Chat allows the user to constrain agent transitions.
- - -## TL;DR -Recently, FSM Group Chat is released that allows the user to input a transition graph to constrain agent transitions. This is useful as the number of agents increases because the number of transition pairs (N choose 2 combinations) increases exponentially increasing the risk of sub-optimal transitions, which leads to wastage of tokens and/or poor outcomes. - -## Possible use-cases for transition graph -1. One-pass workflow, i.e., we want each agent to only have one pass at the problem, Agent A -> B -> C. -2. Decision tree flow, like a decision tree, we start with a root node (agent), and flow down the decision tree with agents being nodes. For example, if the query is a SQL query, hand over to the SQL agent, else if the query is a RAG query, hand over to the RAG agent. -3. Sequential Team Ops. Suppose we have a team of 3 developer agents, each responsible for a different GitHub repo. We also have a team of business analyst that discuss and debate the overall goal of the user. We could have the manager agent of the developer team speak to the manager agent of the business analysis team. That way, the discussions are more focused team-wise, and better outcomes can be expected. - -Note that we are not enforcing a directed acyclic graph; the user can specify the graph to be acyclic, but cyclic workflows can also be useful to iteratively work on a problem, and layering additional analysis onto the solution. - - -## Usage Guide -We have added two parameters `allowed_or_disallowed_speaker_transitions` and `speaker_transitions_type`. -- `allowed_or_disallowed_speaker_transitions`: is a dictionary with the type expectation of `{Agent: [Agent]}`. The key refers to the source agent, while the value(s) in the list refers to the target agent(s). If none, a fully connection graph is assumed. -- `speaker_transitions_type`: is a string with the type expectation of string, and specifically, one of ["allowed", "disallowed"]. We wanted the user to be able to supply a dictionary of allowed or disallowed transitions to improve the ease of use. In the code base, we would invert the disallowed transition into a allowed transition dictionary `allowed_speaker_transitions_dict`. - - -### Application of the FSM Feature - -A quick demonstration of how to initiate a FSM-based `GroupChat` in the `AutoGen` framework. In this demonstration, if we consider each agent as a state, and each agent speaks according to certain conditions. For example, User always initiates the task first, followed by Planner creating a plan. Then Engineer and Executor work alternately, with Critic intervening when necessary, and after Critic, only Planner should revise additional plans. Each state can only exist at a time, and there are transition conditions between states. Therefore, GroupChat can be well abstracted as a Finite-State Machine (FSM). - - - - -### Usage - -0. Pre-requisites -```bash -pip install autogen[graph] -``` - -1. Import dependencies - - ```python - from autogen.agentchat import GroupChat, AssistantAgent, UserProxyAgent, GroupChatManager - from autogen.oai.openai_utils import config_list_from_dotenv - ``` -2. Configure LLM parameters - - ```python - # Please feel free to change it as you wish - config_list = config_list_from_dotenv( - dotenv_file_path='.env', - model_api_key_map={'gpt-4-1106-preview':'OPENAI_API_KEY'}, - filter_dict={ - "model": { - "gpt-4-1106-preview" - } - } - ) - - gpt_config = { - "cache_seed": None, - "temperature": 0, - "config_list": config_list, - "timeout": 100, - } - ``` - -3. Define the task - - ```python - # describe the task - task = """Add 1 to the number output by the previous role. If the previous number is 20, output "TERMINATE".""" - ``` - -4. Define agents - - ```python - # agents configuration - engineer = AssistantAgent( - name="Engineer", - llm_config=gpt_config, - system_message=task, - description="""I am **ONLY** allowed to speak **immediately** after `Planner`, `Critic` and `Executor`. - If the last number mentioned by `Critic` is not a multiple of 5, the next speaker must be `Engineer`. - """ - ) - - planner = AssistantAgent( - name="Planner", - system_message=task, - llm_config=gpt_config, - description="""I am **ONLY** allowed to speak **immediately** after `User` or `Critic`. - If the last number mentioned by `Critic` is a multiple of 5, the next speaker must be `Planner`. - """ - ) - - executor = AssistantAgent( - name="Executor", - system_message=task, - is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("FINISH"), - llm_config=gpt_config, - description="""I am **ONLY** allowed to speak **immediately** after `Engineer`. - If the last number mentioned by `Engineer` is a multiple of 3, the next speaker can only be `Executor`. - """ - ) - - critic = AssistantAgent( - name="Critic", - system_message=task, - llm_config=gpt_config, - description="""I am **ONLY** allowed to speak **immediately** after `Engineer`. - If the last number mentioned by `Engineer` is not a multiple of 3, the next speaker can only be `Critic`. - """ - ) - - user_proxy = UserProxyAgent( - name="User", - system_message=task, - code_execution_config=False, - human_input_mode="NEVER", - llm_config=False, - description=""" - Never select me as a speaker. - """ - ) - ``` - - 1. Here, I have configured the `system_messages` as "task" because every agent should know what it needs to do. In this example, each agent has the same task, which is to count in sequence. - 2. **The most important point is the `description` parameter, where I have used natural language to describe the transition conditions of the FSM. Because the manager knows which agents are available next based on the constraints of the graph, I describe in the `description` field of each candidate agent when it can speak, effectively describing the transition conditions in the FSM.** - -5. Define the graph - - ```python - graph_dict = {} - graph_dict[user_proxy] = [planner] - graph_dict[planner] = [engineer] - graph_dict[engineer] = [critic, executor] - graph_dict[critic] = [engineer, planner] - graph_dict[executor] = [engineer] - ``` - - 1. **The graph here and the transition conditions mentioned above together form a complete FSM. Both are essential and cannot be missing.** - 2. You can visualize it as you wish, which is shown as follows - -  - -6. Define a `GroupChat` and a `GroupChatManager` - - ```python - agents = [user_proxy, engineer, planner, executor, critic] - - # create the groupchat - group_chat = GroupChat(agents=agents, messages=[], max_round=25, allowed_or_disallowed_speaker_transitions=graph_dict, allow_repeat_speaker=None, speaker_transitions_type="allowed") - - # create the manager - manager = GroupChatManager( - groupchat=group_chat, - llm_config=gpt_config, - is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("TERMINATE"), - code_execution_config=False, - ) - ``` - -7. Initiate the chat - - ```python - # initiate the task - user_proxy.initiate_chat( - manager, - message="1", - clear_history=True - ) - ``` - -8. You may get the following output(I deleted the ignorable warning): - - ``` - User (to chat_manager): - - 1 - - -------------------------------------------------------------------------------- - Planner (to chat_manager): - - 2 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 3 - - -------------------------------------------------------------------------------- - Executor (to chat_manager): - - 4 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 5 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 6 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 7 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 8 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 9 - - -------------------------------------------------------------------------------- - Executor (to chat_manager): - - 10 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 11 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 12 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 13 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 14 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 15 - - -------------------------------------------------------------------------------- - Executor (to chat_manager): - - 16 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 17 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 18 - - -------------------------------------------------------------------------------- - Engineer (to chat_manager): - - 19 - - -------------------------------------------------------------------------------- - Critic (to chat_manager): - - 20 - - -------------------------------------------------------------------------------- - Planner (to chat_manager): - - TERMINATE - ``` - -## Notebook examples -More examples can be found in the [notebook](https://microsoft.github.io/autogen/docs/notebooks/agentchat_groupchat_finite_state_machine/). The notebook includes more examples of possible transition paths such as (1) hub and spoke, (2) sequential team operations, and (3) think aloud and debate. It also uses the function `visualize_speaker_transitions_dict` from `autogen.graph_utils` to visualize the various graphs. +--- +title: "FSM Group Chat -- User-specified agent transitions" +authors: + - joshkyh + - freedeaths +tags: [AutoGen] +--- + + +Finite State Machine (FSM) Group Chat allows the user to constrain agent transitions.
+ + +## TL;DR +Recently, FSM Group Chat is released that allows the user to input a transition graph to constrain agent transitions. This is useful as the number of agents increases because the number of transition pairs (N choose 2 combinations) increases exponentially increasing the risk of sub-optimal transitions, which leads to wastage of tokens and/or poor outcomes. + +## Possible use-cases for transition graph +1. One-pass workflow, i.e., we want each agent to only have one pass at the problem, Agent A -> B -> C. +2. Decision tree flow, like a decision tree, we start with a root node (agent), and flow down the decision tree with agents being nodes. For example, if the query is a SQL query, hand over to the SQL agent, else if the query is a RAG query, hand over to the RAG agent. +3. Sequential Team Ops. Suppose we have a team of 3 developer agents, each responsible for a different GitHub repo. We also have a team of business analyst that discuss and debate the overall goal of the user. We could have the manager agent of the developer team speak to the manager agent of the business analysis team. That way, the discussions are more focused team-wise, and better outcomes can be expected. + +Note that we are not enforcing a directed acyclic graph; the user can specify the graph to be acyclic, but cyclic workflows can also be useful to iteratively work on a problem, and layering additional analysis onto the solution. + + +## Usage Guide +We have added two parameters `allowed_or_disallowed_speaker_transitions` and `speaker_transitions_type`. +- `allowed_or_disallowed_speaker_transitions`: is a dictionary with the type expectation of `{Agent: [Agent]}`. The key refers to the source agent, while the value(s) in the list refers to the target agent(s). If none, a fully connection graph is assumed. +- `speaker_transitions_type`: is a string with the type expectation of string, and specifically, one of ["allowed", "disallowed"]. We wanted the user to be able to supply a dictionary of allowed or disallowed transitions to improve the ease of use. In the code base, we would invert the disallowed transition into a allowed transition dictionary `allowed_speaker_transitions_dict`. + + +### Application of the FSM Feature + +A quick demonstration of how to initiate a FSM-based `GroupChat` in the `AutoGen` framework. In this demonstration, if we consider each agent as a state, and each agent speaks according to certain conditions. For example, User always initiates the task first, followed by Planner creating a plan. Then Engineer and Executor work alternately, with Critic intervening when necessary, and after Critic, only Planner should revise additional plans. Each state can only exist at a time, and there are transition conditions between states. Therefore, GroupChat can be well abstracted as a Finite-State Machine (FSM). + + + + +### Usage + +0. Pre-requisites +```bash +pip install autogen[graph] +``` + +1. Import dependencies + + ```python + from autogen.agentchat import GroupChat, AssistantAgent, UserProxyAgent, GroupChatManager + from autogen.oai.openai_utils import config_list_from_dotenv + ``` +2. Configure LLM parameters + + ```python + # Please feel free to change it as you wish + config_list = config_list_from_dotenv( + dotenv_file_path='.env', + model_api_key_map={'gpt-4-1106-preview':'OPENAI_API_KEY'}, + filter_dict={ + "model": { + "gpt-4-1106-preview" + } + } + ) + + gpt_config = { + "cache_seed": None, + "temperature": 0, + "config_list": config_list, + "timeout": 100, + } + ``` + +3. Define the task + + ```python + # describe the task + task = """Add 1 to the number output by the previous role. If the previous number is 20, output "TERMINATE".""" + ``` + +4. Define agents + + ```python + # agents configuration + engineer = AssistantAgent( + name="Engineer", + llm_config=gpt_config, + system_message=task, + description="""I am **ONLY** allowed to speak **immediately** after `Planner`, `Critic` and `Executor`. + If the last number mentioned by `Critic` is not a multiple of 5, the next speaker must be `Engineer`. + """ + ) + + planner = AssistantAgent( + name="Planner", + system_message=task, + llm_config=gpt_config, + description="""I am **ONLY** allowed to speak **immediately** after `User` or `Critic`. + If the last number mentioned by `Critic` is a multiple of 5, the next speaker must be `Planner`. + """ + ) + + executor = AssistantAgent( + name="Executor", + system_message=task, + is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("FINISH"), + llm_config=gpt_config, + description="""I am **ONLY** allowed to speak **immediately** after `Engineer`. + If the last number mentioned by `Engineer` is a multiple of 3, the next speaker can only be `Executor`. + """ + ) + + critic = AssistantAgent( + name="Critic", + system_message=task, + llm_config=gpt_config, + description="""I am **ONLY** allowed to speak **immediately** after `Engineer`. + If the last number mentioned by `Engineer` is not a multiple of 3, the next speaker can only be `Critic`. + """ + ) + + user_proxy = UserProxyAgent( + name="User", + system_message=task, + code_execution_config=False, + human_input_mode="NEVER", + llm_config=False, + description=""" + Never select me as a speaker. + """ + ) + ``` + + 1. Here, I have configured the `system_messages` as "task" because every agent should know what it needs to do. In this example, each agent has the same task, which is to count in sequence. + 2. **The most important point is the `description` parameter, where I have used natural language to describe the transition conditions of the FSM. Because the manager knows which agents are available next based on the constraints of the graph, I describe in the `description` field of each candidate agent when it can speak, effectively describing the transition conditions in the FSM.** + +5. Define the graph + + ```python + graph_dict = {} + graph_dict[user_proxy] = [planner] + graph_dict[planner] = [engineer] + graph_dict[engineer] = [critic, executor] + graph_dict[critic] = [engineer, planner] + graph_dict[executor] = [engineer] + ``` + + 1. **The graph here and the transition conditions mentioned above together form a complete FSM. Both are essential and cannot be missing.** + 2. You can visualize it as you wish, which is shown as follows + +  + +6. Define a `GroupChat` and a `GroupChatManager` + + ```python + agents = [user_proxy, engineer, planner, executor, critic] + + # create the groupchat + group_chat = GroupChat(agents=agents, messages=[], max_round=25, allowed_or_disallowed_speaker_transitions=graph_dict, allow_repeat_speaker=None, speaker_transitions_type="allowed") + + # create the manager + manager = GroupChatManager( + groupchat=group_chat, + llm_config=gpt_config, + is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("TERMINATE"), + code_execution_config=False, + ) + ``` + +7. Initiate the chat + + ```python + # initiate the task + user_proxy.initiate_chat( + manager, + message="1", + clear_history=True + ) + ``` + +8. You may get the following output(I deleted the ignorable warning): + + ``` + User (to chat_manager): + + 1 + + -------------------------------------------------------------------------------- + Planner (to chat_manager): + + 2 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 3 + + -------------------------------------------------------------------------------- + Executor (to chat_manager): + + 4 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 5 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 6 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 7 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 8 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 9 + + -------------------------------------------------------------------------------- + Executor (to chat_manager): + + 10 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 11 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 12 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 13 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 14 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 15 + + -------------------------------------------------------------------------------- + Executor (to chat_manager): + + 16 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 17 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 18 + + -------------------------------------------------------------------------------- + Engineer (to chat_manager): + + 19 + + -------------------------------------------------------------------------------- + Critic (to chat_manager): + + 20 + + -------------------------------------------------------------------------------- + Planner (to chat_manager): + + TERMINATE + ``` + +## Notebook examples +More examples can be found in the [notebook](https://microsoft.github.io/autogen/docs/notebooks/agentchat_groupchat_finite_state_machine/). The notebook includes more examples of possible transition paths such as (1) hub and spoke, (2) sequential team operations, and (3) think aloud and debate. It also uses the function `visualize_speaker_transitions_dict` from `autogen.graph_utils` to visualize the various graphs. diff --git a/website/blog/authors.yml b/website/blog/authors.yml index b52fffbdd0fa..70a4e5c0f9c1 100644 --- a/website/blog/authors.yml +++ b/website/blog/authors.yml @@ -1,125 +1,125 @@ -sonichi: - name: Chi Wang - title: Principal Researcher at Microsoft Research - url: https://www.linkedin.com/in/chi-wang-49b15b16/ - image_url: https://github.com/sonichi.png - -qingyunwu: - name: Qingyun Wu - title: Assistant Professor at the Pennsylvania State University - url: https://qingyun-wu.github.io/ - image_url: https://github.com/qingyun-wu.png - -yiranwu: - name: Yiran Wu - title: PhD student at Pennsylvania State University - url: https://github.com/kevin666aa - image_url: https://github.com/kevin666aa.png - -jialeliu: - name: Jiale Liu - title: Undergraduate student at Xidian University - url: https://leoljl.github.io - image_url: https://github.com/LeoLjl/leoljl.github.io/blob/main/profile.jpg?raw=true - -thinkall: - name: Li Jiang - title: Senior Software Engineer at Microsoft - url: https://github.com/thinkall - image_url: https://github.com/thinkall.png - -rickyloynd-microsoft: - name: Ricky Loynd - title: Senior Research Engineer at Microsoft - url: https://github.com/rickyloynd-microsoft - image_url: https://github.com/rickyloynd-microsoft.png - -samershi: - name: Saleema Amershi - title: Senior Principal Research Manager at Microsoft Research - url: https://github.com/samershi - image_url: https://github.com/samershi.png - -pcdeadeasy: - name: Piali Choudhury - title: Principal RSDE at Microsoft Research - url: https://github.com/pcdeadeasy - image_url: https://github.com/pcdeadeasy.png - -victordibia: - name: Victor Dibia - title: Principal RSDE at Microsoft Research - url: https://github.com/victordibia - image_url: https://github.com/victordibia.png - -afourney: - name: Adam Fourney - title: Principal Researcher Microsoft Research - url: https://www.adamfourney.com - image_url: https://github.com/afourney.png - -beibinli: - name: Beibin Li - title: Senior Research Engineer at Microsoft - url: https://github.com/beibinli - image_url: https://github.com/beibinli.png - -gagb: - name: Gagan Bansal - title: Senior Researcher at Microsoft Research - url: https://www.linkedin.com/in/gagan-bansal/ - image_url: https://github.com/gagb.png - -jieyuz2: - name: Jieyu Zhang - title: PhD student at University of Washington - url: https://jieyuz2.github.io/ - image_url: https://github.com/jieyuz2.png - -julianakiseleva: - name: Julia Kiseleva - title: Senior Researcher at Microsoft Research - url: https://github.com/julianakiseleva/ - image_url: https://avatars.githubusercontent.com/u/5908392?v=4 - -narabzad: - name: Negar Arabzadeh - title: PhD student at the University of Waterloo - url: https://www.negara.me/ - image_url: https://github.com/Narabzad.png - -LinxinS97: - name: Linxin Song - title: MS student at Waseda University - url: https://linxins97.github.io/ - image_url: https://github.com/LinxinS97.png - -skzhang1: - name: Shaokun Zhang - title: PhD student at the Pennsylvania State University - url: https://github.com/skzhang1 - image_url: https://github.com/skzhang1.png - -olgavrou: - name: Olga Vrousgou - title: Senior Software Engineer at Microsoft Research - url: https://github.com/olgavrou/ - image_url: https://github.com/olgavrou.png - -joshkyh: - name: Joshua Kim - title: AI Freelancer at SpectData - url: https://github.com/joshkyh/ - image_url: https://github.com/joshkyh.png - -freedeaths: - name: Yishen Sun - title: Data Scientist at PingCAP LAB - url: https://github.com/freedeaths/ - image_url: https://github.com/freedeaths.png - -yifanzeng: - name: Yifan Zeng - title: PhD student at Oregon State University - url: https://xhmy.github.io/ +sonichi: + name: Chi Wang + title: Principal Researcher at Microsoft Research + url: https://www.linkedin.com/in/chi-wang-49b15b16/ + image_url: https://github.com/sonichi.png + +qingyunwu: + name: Qingyun Wu + title: Assistant Professor at the Pennsylvania State University + url: https://qingyun-wu.github.io/ + image_url: https://github.com/qingyun-wu.png + +yiranwu: + name: Yiran Wu + title: PhD student at Pennsylvania State University + url: https://github.com/kevin666aa + image_url: https://github.com/kevin666aa.png + +jialeliu: + name: Jiale Liu + title: Undergraduate student at Xidian University + url: https://leoljl.github.io + image_url: https://github.com/LeoLjl/leoljl.github.io/blob/main/profile.jpg?raw=true + +thinkall: + name: Li Jiang + title: Senior Software Engineer at Microsoft + url: https://github.com/thinkall + image_url: https://github.com/thinkall.png + +rickyloynd-microsoft: + name: Ricky Loynd + title: Senior Research Engineer at Microsoft + url: https://github.com/rickyloynd-microsoft + image_url: https://github.com/rickyloynd-microsoft.png + +samershi: + name: Saleema Amershi + title: Senior Principal Research Manager at Microsoft Research + url: https://github.com/samershi + image_url: https://github.com/samershi.png + +pcdeadeasy: + name: Piali Choudhury + title: Principal RSDE at Microsoft Research + url: https://github.com/pcdeadeasy + image_url: https://github.com/pcdeadeasy.png + +victordibia: + name: Victor Dibia + title: Principal RSDE at Microsoft Research + url: https://github.com/victordibia + image_url: https://github.com/victordibia.png + +afourney: + name: Adam Fourney + title: Principal Researcher Microsoft Research + url: https://www.adamfourney.com + image_url: https://github.com/afourney.png + +beibinli: + name: Beibin Li + title: Senior Research Engineer at Microsoft + url: https://github.com/beibinli + image_url: https://github.com/beibinli.png + +gagb: + name: Gagan Bansal + title: Senior Researcher at Microsoft Research + url: https://www.linkedin.com/in/gagan-bansal/ + image_url: https://github.com/gagb.png + +jieyuz2: + name: Jieyu Zhang + title: PhD student at University of Washington + url: https://jieyuz2.github.io/ + image_url: https://github.com/jieyuz2.png + +julianakiseleva: + name: Julia Kiseleva + title: Senior Researcher at Microsoft Research + url: https://github.com/julianakiseleva/ + image_url: https://avatars.githubusercontent.com/u/5908392?v=4 + +narabzad: + name: Negar Arabzadeh + title: PhD student at the University of Waterloo + url: https://www.negara.me/ + image_url: https://github.com/Narabzad.png + +LinxinS97: + name: Linxin Song + title: MS student at Waseda University + url: https://linxins97.github.io/ + image_url: https://github.com/LinxinS97.png + +skzhang1: + name: Shaokun Zhang + title: PhD student at the Pennsylvania State University + url: https://github.com/skzhang1 + image_url: https://github.com/skzhang1.png + +olgavrou: + name: Olga Vrousgou + title: Senior Software Engineer at Microsoft Research + url: https://github.com/olgavrou/ + image_url: https://github.com/olgavrou.png + +joshkyh: + name: Joshua Kim + title: AI Freelancer at SpectData + url: https://github.com/joshkyh/ + image_url: https://github.com/joshkyh.png + +freedeaths: + name: Yishen Sun + title: Data Scientist at PingCAP LAB + url: https://github.com/freedeaths/ + image_url: https://github.com/freedeaths.png + +yifanzeng: + name: Yifan Zeng + title: PhD student at Oregon State University + url: https://xhmy.github.io/ image_url: https://xhmy.github.io/assets/img/photo.JPG