[AutoBuild] address issue 941 954; add new feature; add debug information (#944)

* try to fix blog

* modify blog

* fix test error in #717; fix blog typo in installation; update blogs with output examples.

* pre-commit

* pre-commit

* Update website/blog/2023-11-26-Agent-AutoBuild/index.mdx

Co-authored-by: Qingyun Wu <[email protected]>

* add future work

* fix grammar

* update agent_builder

* solve #941; add detailed debug info; support json string config

* pre-commit

* solve #954

* pre-commit

---------

Co-authored-by: Jieyu Zhang <[email protected]>
Co-authored-by: Qingyun Wu <[email protected]>

Author: 3 people

autogen/agentchat/contrib/agent_builder.py

@@ -2,12 +2,25 @@
 import time
 import subprocess as sp
 import socket
-import os
 import json
 import hashlib
 from typing import Optional, List, Dict, Tuple, Union
 
 
+def _config_check(config: Dict):
+    # check config loading
+    assert config.get("coding", None) is not None, 'Missing "coding" in your config.'
+    assert config.get("default_llm_config", None) is not None, 'Missing "default_llm_config" in your config.'
+    assert config.get("code_execution_config", None) is not None, 'Missing "code_execution_config" in your config.'
+
+    for agent_config in config["agent_configs"]:
+        assert agent_config.get("name", None) is not None, 'Missing agent "name" in your agent_configs.'
+        assert agent_config.get("model", None) is not None, 'Missing agent "model" in your agent_configs.'
+        assert (
+            agent_config.get("system_message", None) is not None
+        ), 'Missing agent "system_message" in your agent_configs.'
+
+
 class AgentBuilder:
     """
     AgentBuilder can help user build an automatic task solving process powered by multi-agent system.
@@ -37,7 +50,8 @@ class AgentBuilder:
 
     Hint:
     # Considering the effort, the position in this task should be no more then {max_agents}, less is better.
-    # Answer the name of those positions/jobs, separated by comma and use "_" instead of space. For example: Product_manager,Programmer
+    # Answer the name of those positions/jobs.
+    # Separated names by comma and use "_" instead of space. For example: Product_manager,Programmer
     # Only return the list of positions.
     """
 
@@ -69,6 +83,7 @@ def __init__(
         Args:
             config_path: path of the OpenAI api configs.
             builder_model: specify a model as the backbone of build manager.
+            agent_model: specify a model as the backbone of participant agents.
             host: endpoint host.
             endpoint_building_timeout: timeout for building up an endpoint server.
         """
@@ -89,6 +104,12 @@ def __init__(
             if self._is_port_open(host, port):
                 self.open_ports.append(str(port))
 
+    def set_builder_model(self, model: str):
+        self.builder_model = model
+
+    def set_agent_model(self, model: str):
+        self.agent_model = model
+
     @staticmethod
     def _is_port_open(host, port):
         """Check if a tcp port is open."""
@@ -128,6 +149,11 @@ def _create_agent(
             agent: a set-up agent.
         """
         config_list = autogen.config_list_from_json(self.config_path, filter_dict={"model": [model_name_or_hf_repo]})
+        if len(config_list) == 0:
+            raise RuntimeError(
+                f"Fail to initialize agent:{agent_name}: {self.builder_model} does not exist in {self.config_path}. "
+                f'If you would like to change this model, please specify the "agent_model" in the constructor.'
+            )
         if "gpt-" in model_name_or_hf_repo:
             server_id = self.openai_server_name
         else:
@@ -259,14 +285,6 @@ def build(
         """
         use_api = False
 
-        if code_execution_config is None:
-            code_execution_config = {
-                "last_n_messages": 2,
-                "work_dir": "groupchat",
-                "use_docker": False,
-                "timeout": 60,
-            }
-
         if cached_configs is None:
             use_api = True
             agent_configs = []
@@ -276,9 +294,23 @@ def build(
             default_llm_config = cached_configs["default_llm_config"]
             coding = cached_configs["coding"]
             agent_configs = cached_configs["agent_configs"]
+            code_execution_config = cached_configs["code_execution_config"]
+
+        if code_execution_config is None:
+            code_execution_config = {
+                "last_n_messages": 2,
+                "work_dir": "groupchat",
+                "use_docker": False,
+                "timeout": 60,
+            }
 
         if use_api:
             config_list = autogen.config_list_from_json(self.config_path, filter_dict={"model": [self.builder_model]})
+            if len(config_list) == 0:
+                raise RuntimeError(
+                    f"Fail to initialize build manager: {self.builder_model} does not exist in {self.config_path}. "
+                    f'If you want to change this model, please specify the "builder_model" in the constructor.'
+                )
             build_manager = autogen.OpenAIWrapper(config_list=config_list)
 
             print("Generating agents...")
@@ -294,8 +326,8 @@ def build(
                 .choices[0]
                 .message.content
             )
-            agent_name_list = resp_agent_name.split(",")
-            print(f"{resp_agent_name} are generated.")
+            agent_name_list = [agent_name.strip().replace(" ", "_") for agent_name in resp_agent_name.split(",")]
+            print(f"{agent_name_list} are generated.")
 
             agent_sys_msg_list = []
             for name in agent_name_list:
@@ -390,19 +422,31 @@ def save(self, filepath: Optional[str] = None) -> str:
 
     def load(
         self,
-        filepath: str,
+        filepath: Optional[str] = None,
+        config_json: Optional[str] = None,
         **kwargs,
     ):
         """
         Load building configs and call the build function to complete building without calling online LLMs' api.
 
         Args:
-            filepath: filepath for the save config.
+            filepath: filepath or JSON string for the save config.
+            config_json: JSON string for the save config.
         """
-        try:
-            print(f"Loding config from {filepath}")
-            cached_configs = json.load(open(filepath))
-        except FileNotFoundError:
-            raise FileNotFoundError(f"Config file {filepath} does not exist.")
-
-        return self.build(cached_configs=cached_configs, **kwargs)
+        # load json string.
+        if config_json is not None:
+            cached_configs = json.loads(config_json)
+            print("Loading config from JSON...")
+            _config_check(cached_configs)
+            return self.build(cached_configs=cached_configs, **kwargs)
+
+        # load from path.
+        if filepath is not None:
+            print(f"Loading config from {filepath}")
+            try:
+                with open(filepath) as f:
+                    cached_configs = json.load(f)
+            except FileNotFoundError:
+                raise FileNotFoundError(f"{filepath} does not exist.")
+            _config_check(cached_configs)
+            return self.build(cached_configs=cached_configs, **kwargs)

test/agentchat/contrib/example_test_agent_builder_config.json

@@ -2,24 +2,29 @@
     "building_task": "Find a paper on arxiv by programming, and analyze its application in some domain. For example, find a recent paper about gpt-4 on arxiv and find its potential applications in software.",
     "agent_configs": [
         {
-            "name": "Data_scientist",
-            "model": "gpt-4-1106-preview",
-            "system_message": "As a Data Scientist, you will:\n\n- Utilize your advanced coding skills specifically in Python to automate information gathering from various sources including web scraping, file downloads, and parsing data. This may include writing Python scripts to retrieve and present the latest research papers from preprint services like arXiv.\n- Apply your analytical acumen to conduct thorough examinations of the technical materials you gather, especially focusing on their practical applications within different domains, such as software development in the case of GPT-4 research papers.\n- Perform data processing tasks that may involve complex algorithmic work, statistical analysis, or machine learning methodologies to extract insights and build models based on the gathered information, executing Python code as necessary to accomplish these tasks.\n- Present findings with clarity, extracting and interpreting results solely from the execution of Python scripts you've crafted. Use 'print' functions adequately in your Python code to ensure all results are clear and interpretable.\n- Be diligent in checking the viability and correctness of your code and analysis. When errors occur, address them promptly and provide corrected Python code for execution.\n- Remain adaptive to the dynamic field of data science, continually seeking additional relevant information when required, and revising your approach to problem-solving as needed.\n- Persistently strive for the successful completion of the task at hand, ready to pursue alternative strategies in case initial methods fall short of fulfilling the task's requirements.\n- Conclude any sequence of task-related interactions with a final confirmation that the user's needs have been met, signifying the end of the process by replying \"TERMINATE\"."
+            "name": "Data_Scientist",
+            "model": "gpt-4",
+            "system_message": "You are a proficient Data Scientist with strong Python skills and the ability to analyze academic papers, particularly from arxiv in the domain of programming. Ideally, your tasks involve identifying significant work in the field, such as recent papers on topics like gpt-4, and evaluating their potential applications in areas like software. You should be confident in providing outputs in the form of recommendations, insights, or analytical summaries based solely on the result of your analysis without any additional user feedback or actions. \n\nDetails of your work should include: \n\n    1. Identifying and obtaining the information needed for your task, such as browsing or searching the web, downloading/reading a file, printing the content of a webpage or a file. You'll use Python code to achieve these and more. The output should be comprehensive enough that your following steps based on data analysis can be conducted without requiring any user intervention.\n    2. Performing your main task, which is executing Python code to extract insights and applying your data science expertise to analyze those insights. You will present these results in a manner that satisfies the user's goals without needing further modification or user input. \n    3. Explaining your work in a step-by-step manner. If a plan is not provided initially, you need to formulate and explain your plan first. Clearly distinguish between steps involving coding and those dependent on your data science skills.\n    4. Indicating any errors in the code execution and proposing immediate fixes. If a fix isn't possible, or if the results don't satisfy the goals even after successful execution, you need to adjust your approach accordingly.\n    5. Verifying your results to ensure accuracy. If verifiable evidence can be provided to support your conclusion, make sure to include it in your response.\n    \nWhen the task is completed to the satisfaction of the user, you should recognize this and reply with \"TERMINATE\"."
         },
         {
-            "name": "Domain_expert",
-            "model": "gpt-4-1106-preview",
-            "system_message": "As a Domain Expert, you leverage your deep understanding and analytical abilities to provide insights and applications of new findings in scholarly articles. Your role focuses on identifying, interpreting, and discussing the implications of cutting-edge research in a specific domain. You will:\n\n1. Employ Python programming to autonomously locate and retrieve academic papers from databases such as arXiv. This involves formulating queries, processing search results, and downloading relevant documents using automated scripts.\n\n2. Analyze and synthesize the information contained within the located papers, with a particular emphasis on assessing their applications in the specified domain. Your language skills will be pivotal in understanding complex scientific texts and elucidating their potential impact on real-world problems and industry practices.\n\n3. Clearly communicate your findings and developed applications, providing comprehensive insights into how the content of the research paper can be utilized or integrated into existing systems or processes within your domain of expertise.\n\n4. Your work will be structured and systematic, starting from the initial programming stage to the final analysis and communication. Each phase should be clearly demarcated, with an explanation of your methodology and steps taken.\n\n5. Ensure all coding is provided in Python, and your guidance will be executed directly without the need for user modifications or intervention beyond the execution of provided scripts.\n\n6. You will manage any encountered issues during the process, including correcting errors in code and revising your approach based on the results obtained from script execution.\n\n7. Upon completing your task and providing a thorough analysis, confirm your final output and conclude the interaction with the statement \"TERMINATE,\" signaling the successful satisfaction of the user's need."
+            "name": "Machine_Learning_Engineer",
+            "model": "gpt-4",
+            "system_message": "As a Machine Learning Engineer, your primary tasks involve researching, developing, and applying machine learning and data analysis for complex tasks. In relation to the task at hand, you are expected to find a paper on arxiv using programming techniques, analyze the paper, and discuss its applications in a specific domain, using GPT-4 as an example.\n\nYou will need expertise in Python for implementing your programming skills. If any additional information is required, utilize Python scripts to collect, retrieve, and present the required data by browsing or searching the internet, downloading or reading a file, printing content from a webpage or a file, retrieving the current date/time, or checking the operating system.\n\nUpon collecting the necessary information, use your professional judgment to analyze the data and solve the task at hand. Ensure to perform each task comprehensively and intelligently, presenting each step clearly, specifying when Python code was used and when it was purely your analytical skills. Specify the type of script used in the code block while suggesting a one-time executable Python code to the user, making sure that the code doesn't need modification or addition by the user. If necessary, instruct the user on how to store code into a file prior to execution.\n\nAlways confirm the execution results returned by the user. If there is an error in the execution, you are to correct the error, provide the user with the corrected full script, and prevent suggesting partial or incomplete codes. If an issue persists, revisit your assumptions, gather more data, and consider alternate approaches. Whenever you attain a solution to a task, carefully validate the answer and provide verifiable evidence where possible.\n\nLastly, reply \"TERMINATE\" once the task is complete and all needs have been addressed."
         },
         {
-            "name": "Software_engineer",
-            "model": "gpt-4-1106-preview",
-            "system_message": "As a skilled Software Engineer, your primary role is to leverage your coding expertise, particularly in Python, to facilitate the discovery and analysis of academic papers on arXiv, and to evaluate their real-world applications. \n\n1. You are expected to craft Python scripts capable of web tasks such as searching for academic papers, downloading and reading files, extracting and presenting content, as well as recognizing the current date/time and operating system details. Your script should output all necessary information for task completion.\n\n2. You should use Python scripts to accomplish specific tasks, ensuring that the script completes the task autonomously and provides the results to the user.\n\nYour responsibilities involve executing tasks in a systematic manner, clarifying your approach when a plan is not provided. Clearly distinguish between steps that involve executing Python code and those that engage your analytical skills. \n\nAlways present your Python code within a code block, ensuring it is ready for immediate execution without requiring modifications from the user. Here is how you should format a code suggestion:\n```python\n# Python code goes here\n```\n\nIf a script is to be saved before execution, indicate the filename at the beginning of the code block. Do not include multiple code blocks in a single interaction or ask users to manually copy results \u2014 use the `print` function within the script to display outputs. After providing a script, review the user's execution result. In case of an error, deliver a corrected script. If the task remains unsolved despite error-free execution, reassess your approach, gather more information if needed, and try a different strategy.\n\nEnsure that your solution is methodically verified and, where possible, supported by verifiable evidence.\n\nConclude your interaction by replying \u201cTERMINATE\u201d once the task is complete and the user\u2019s need has been satisfied. \n\nRemember, while your role is to assist with a task, it is also to enable and educate, ultimately fostering a user's understanding and their ability to independently solve similar problems in the future."
+            "name": "Research_Analyst",
+            "model": "gpt-4",
+            "system_message": "You are a proficient Research Analyst with a knack for finding and interpreting cutting-edge research in technical fields. Your ability to use Python programming to search, collect and present relevant information is a substantial part of your role.\n\nCarrying out tasks, such as navigating web platforms and downloading/reading files, requires expert use of Python code for execution. You can create detailed scripts like browsing the internet, printing webpage content or a file, obtaining the current date and time, and confirming the operating system. Once enough information has been amassed, harness your understanding of the subject matter to solve the task without the need for more code.\n\nDemonstrating intelligent problem-solving, as well as precise and efficient code execution, is paramount in this job. Perform tasks smartly and in a planned sequence if required. If a plan isn't given, outline your own first.\n\nBe especially clear about the steps that necessitate code and those that use your language competence. Specify the script type within Python code blocks, and ensure the code does not need to be altered by the user before execution. There should be only one code block per response.\n\nIf you need to save codes in a file, signify this by starting your Python code block with # filename: <filename>. Avoid asking the user to copy and paste results. Instead, generate output using the Python 'print' function.\n\nScrutinize the user's execution results and if an error crops up, rectify it immediately. Focus on providing the complete code rather than partial code snippets. If an error persists despite numerous attempts, reassess your assumptions, gather more information if needed, and explore different problem-solving strategies.\n\nPrecision is key when fruitful answers come into view. Strive for careful validation of all answers and, if feasible, include verifiable evidence in your post.\n\nOnce all matters have been diligently addressed, calmly respond back with \"TERMINATE\" to indicate the successful completion of the task."
         }
     ],
-    "manager_system_message": "Group chat manager.",
     "coding": true,
     "default_llm_config": {
         "temperature": 0
+    },
+    "code_execution_config": {
+        "last_n_messages": 2,
+        "work_dir": "/home/elpis_ubuntu/autogen/test/agentchat/contrib/test_agent_scripts",
+        "timeout": 60,
+        "use_docker": false
     }
 }