Possible Regression - select_speaker failed to resolve the next speaker's name - #842
Comments
|
@afourney FYI |
|
Interesting. I'll add it to the speaker selection tests and see if I can replicate it. It's unclear at which stage this is happening. One of my major test cases (human eval with a third agent) uses underscores in the agent name without any issue. So I am skeptical it comes from GroupChat itself. Maybe the model is doing it? |
|
So it seems there are several issues. I am trying to replicate the notebook agentchat_groupchat_RAG.ipynb into a normal python program with other models.
|
|
any update on this? I try running agentchat_groupchat.ipynb, and encounters the same issue:
|
|
same |
|
Pretty much the same thing happens when I want to use Groupchat. In the beginning, something like the following is sometimes generated "←[33mContentCreator←[0m (to chat_manager):". A friend saw the following on his screen : "\033[34m" and "\033[0m". These look like colour codes? And then it goes all wrong, because the Groupchat manager cannot find the next speaker, and then further generates "GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:" without a speaker, while there is still some output from a speaker, but from which speaker is not clear. Until this bug (and related issues) gets fixed, Autogen is pretty much worthless. |
|
same issue, cleaning up the cache at every run does not resolve the issue. |
|
same issue I am not sure what to report here to help solve the issue. I am running four agents, and using LM studio as the server. It looks like the agents chat either break off, requestion a prompt or it loops around getting similar responses. I don't know if this helps - say this is the model: { and this is a snip of the chat: It looks like the UserproxyAgent gets a somewhat "confusing" pre_promp: "You are in a role play game. The following roles are available:\nDirector:" .... and then the Director is prompted to select: "Then select the next role from ['Director', 'Assistant', 'Critic', 'AgentBuilder'] to play. Only return the role." I wonder if a change to the prepompt texst would help? [](https://discord.com/channels/1110598183144399058/1177683199514660934/1187733442977337365 |
|
@AssetDK Link to discord channel is not correct I think (resulting in no text channels). |
|
I'm hitting this problem also. |
|
Hi everyone. I'm looking into this this now. Notably, I would expect GroupChat to work poorly on everything but GPT-4. It looks like at least on example is using Mistral. I'm not sure what models everyone else is using -- please include this information. @AssetDK as for the confusing "pre-prompt" -- I 100% agree. I tried to have that fixed a while back, but made the mistake of including it in another PR, then removing it so that review would go more smoothly: #736 (comment) I'll follow-through and submit the prompt fix as a standalone PR. I am also a little confused by this pre-prompt? "pre_prompt": "You are in a role play game. The following roles are available:\nDirector: You are the director of our team. The goal of the team is to build a project specific prompt ..\n\nRead the following conversation.\nThen select the next role from ['Director', 'Assistant', 'Critic', 'AgentBuilder'] to play. Only return the role.", There should be at least 4 agents listed, not just the Director. Can you provide an example of how you initialize your agents and your GroupChat? In particular, are any of your agents using function calling? |
Really appreciate this! Im running LM studio version with TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_S.gguf Below you find is my entire code - I tried to add an extra agent "Builder" a meta-agent that I hope can suggest new agent definitions. These tree/four agents should illustrate how I could use the agents to explore what the LLM knows about some EU legislation. I have some ideas for MUCH more interesting teams, but that will have to wait... :-) I wonder, as I am not toooo familiar with Python syntax, if the "extra" commas in the parameter lists could mess things up |
|
|
Hi Per Feldvoss Olsen @AssetDK <https://github.com/AssetDK>
The link is still resulting in “NO TEXT CHANNELS”.
Van: Per Feldvoss Olsen ***@***.***>
Verzonden: jeudi 28 décembre 2023 15:04
Aan: microsoft/autogen ***@***.***>
CC: Maxime-Charmasson ***@***.***>; Comment ***@***.***>
Onderwerp: Re: [microsoft/autogen] Possible Regression - select_speaker failed to resolve the next speaker's name - (Issue #842)
@AssetDK <https://github.com/AssetDK> Link to discord channel is not correct I think (resulting in no text channels).
Perhaps this works better - will edit the original message!
https://discord.com/channels/1110598183144399058/1177683199514660934/1187733442977337365
—
Reply to this email directly, view it on GitHub <#842 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVBNRQTZ3TOHM52FGOXQNCLYLV34DAVCNFSM6AAAAABAEBLW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZRGIYDKNZSGE> .
You are receiving this because you commented. <https://github.com/notifications/beacon/AVBNRQTWSX7VEDVQANOCCR3YLV34DA5CNFSM6AAAAABAEBLW5KWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTPRBKVS.gif> Message ID: ***@***.*** ***@***.***> >
|
|
@afourney - When I disregard the original problem it looks like things are working "somewhat"... until I get this Trackback message, I guess that the code could be fragile if it depends on the reply syntax and try to convert in to messages. |
|
@Maxime-Charmasson well it (the link) works for me, you can find this under Discord for LM studio feature request / support open ai 1.0 API |
|
I understand it is expected to work better with GPT4, but is it possible to confirm how the next player should be written (with escape characters, ...). One "bad" trick could be to match it using fuzzy matching, so we go around little change (such as plurial/singular...), but I am still confused as the next player looks ok for me and we still have an issue to invalidate dirty cache? |
|
Thanks for the additional information. GroupChat is already doing a somewhat fuzzy match by counting how many times each agent is named in response, regardless of formatting: https://github.com/microsoft/autogen/blob/30daa1eac52f8d19f84df5c67cccbab0f9386463/autogen/agentchat/groupchat.py#L267C4-L288C24 It's possible there is a bug here, but I'd love to see the inputs and output to replicate. As for the cache, the cache key should already consider the llm_config settings, so I'm not sure what's going on there. Again, it is possible that there is a bug with that, but it would impact things more broadly... not just group chat: https://github.com/microsoft/autogen/blob/30daa1eac52f8d19f84df5c67cccbab0f9386463/autogen/oai/openai_utils.py#L52-L71C46 I'll keep digging, but the biggest concern I'm seeing is that the speaker selection pre-prompt is missing several agent descriptions in the trace above. However, the system_prompts/descriptions don't match the configuration that was later provided, so something doesn't quite line up. Ideally, what I would like to see is the configuration + the selection pre-prompts of the same run. We can then determine if there is a bug there. The pre-prompt descriptions are generated here: autogen/autogen/agentchat/groupchat.py Lines 253 to 265 in 30daa1e |
|
For starters I think i found the reason this does not work, but I may also have some other basic problem. We often get this This is how the code should actually run, but in some cases the director did not yield such a message. I dont know it this is the root cause but i did not set the names correct: The other thing is that I would think that searching the agents names/roles may be corrupted or "mixed up" when the name is actually a role - that is you should perhaps call the Director agent "joe" - and we should not use the common terms "Director", "Assistant", "Programmer" etc. - even "DirectorA" will be better... NLP is not easy to process programmatically. This is debug from groupchat.py Sorry for the unfinnised message got to run |
|
@AssetDK thanks again for working through this. I really do believe that a good chunk of the problem here is that the core prompting is tuned to GPT-4, and that's breaking speaker selection. The LLM is clearly not following even the basic formatting instructions for speaker selection (but we should be robust to simple formatting issues). One thing that may help is adding a description field to each of your agents. Keep the descriptions short, sweet, and 3rd person. More on descriptions here: https://microsoft.github.io/autogen/blog/2023/12/29/AgentDescriptions |
|
Thanks a lot Adam I have to read all those blog posts! Use caseI will use this for some specific ideas for making the agent produce higher quality output in different ways. I don't know yet if that can be implemented as finetuning or in some way of training. For prof of concept it will have to be done via memory, description or may be even via the prompt. As an example I might need to "make" an expert in a specific branch of philosophy, but I would find it hard to keep that short and sweet... especially if I wanted to have to describe several such experts in distinct ways... that could evolve to long and complicated descriptions. Setup and Teachable agentAlso I need much more knowledge about setting up the environment, I have no idea about what versions I'm running on for any of the software - line numbers are deferent between our code, so we may not be running on the same versions. I just now found this tutorial that is explaining the general setup for VS code (and teachable agents! ... something I will Seed vs memoryFinally, it looks like the version I am running on uses a seed (see below) to preserve some kind of memory I got no idea how that works and how to reset? (teachable agents sounds better, so i will probably need to test that out!) I wonder if Teachable agent is supplementing or replacing memgpt? - and all of this should probably be in the diskussions.. |
Autogen's teachable agents are related to MemGPT in using a vector db for memory that gets constantly updated. But their details and capabilities are quite different. |
The description parameter works fine, however I needed to fill out the system_message parameter for my "Director" and looks like all of the agents are running under a 'user' role even my "Director" - I was under the impression that UserProxyAgent would be the mannager of what agent was next. ... and I still get the Also I will make a new discussion regarding how the team set-up is working. Agent Descriptions (version 0.2.2 onwards) and team setup. #1109 Examples of the message format:Example of text and code reply:Step 3: Develop the AssistantAgent agents Plan: Develop the AssistantAgent team using Python, integrating NLP libraries such as spaCy or NLTK and databases to store and access CAP information. # filename: assistant_agents.py
python -m venv cap_assistant_venv
source cap_assistant_venv/bin/activate
pip install spacy nltk sqlite3
git clone https://github.com/explosion/spaCy.git
cd spaCy && python setup.py install --user
deactivate |
|
First of all, Happy New Year to everyone, and adding my 2 cents in here as well. I would like to the authors at Microsoft and Seehiong for posting workable examples. I too received the same thing but everything else works fine as for the group chat conversation from what I can see so far and I cannot get the user_proxy to execute code in WSL2 Ubuntu 22.04 but I don't think that is related to this issue at all. I agree with @afourney that it's the dependency built into the OpenAI GPT LLMs but why should we have to to also teach the agents who the agents are when in my code, it clearly lists the names of the agents, which like @AssetDK says, is also the roles as well? If that is the case, then how to do we implement teachability to the group chat manager immediately after builder builds all the agents, because based on the code below, they start talking as soon as the function "start_task" completes. This is 100% reproducible from the code below. Thank you, Seehiong, for the great tutorial. Here's my setup and code below: windows 11 WSL2 Ubuntu 22.04 Code base: |
|
im not sure if this is the correct way to solve the problem but i use the following and have no problems |
|
Round robin will definitely solve it, but then the order is fixed -- it just goes round and round in the order you specified. Is that what you want for your scenario? |
|
Providing an update, I was able to get it to work using "speaker_selection_method", the only one that did not work was for "auto".
for the code: |
|
I think this may be a different issue, but I get the same error as @AssetDK replicating the code generation tutorial.
Running the following with llama2 13B: import autogen
config_list = [
{
"base_url": "http://localhost:1234/v1",
"api_key": "NULL"
}
]
llm_config = {
"timeout": 600,
"seed": 40,
"config_list": config_list,
"temperature": 0
}
# create an AssistantAgent named "assistant"
assistant = autogen.AssistantAgent(
name="assistant",
llm_config={
"cache_seed": 55, # seed for caching and reproducibility
"config_list": config_list, # a list of OpenAI API configurations
"temperature": 0, # temperature for sampling
}, # configuration for autogen's enhanced inference API which is compatible with OpenAI API
)
# create a UserProxyAgent instance named "user_proxy"
user_proxy = autogen.UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=10,
is_termination_msg=lambda x: x.get("content", "").rstrip().endswith("TERMINATE"),
code_execution_config={
"work_dir": "coding",
"use_docker": False, # set to True or image name like "python:3" to use docker
},
)
# the assistant receives a message from the user_proxy, which contains the task description
user_proxy.initiate_chat(
assistant,
message="""What date is today? Compare the year-to-date gain for META and TESLA.""",
)Can open separate issue for this if hasn't been addressed here. |
|
@matsuobasho I've seen this error mentioned a few times. E.g., #279 (comment) If you share a trace we can debug further. |
|
@afourney , sure here is the traceback: |
|
@afourney To me it looks like the A simple temporary solution would be to simply do this for groupchat.py line 342:
An empty message will be sent to the other agents - so for that I made the same exception: 
This does not solve the original problem where we do not get the next speaker... but the chat now runs and runs! |
|
has anyone figured out what the problem or solution is here? it's definitely speaker selection, but it's not clear to me why it's happening. |
|
project was abandoned. they did some hype to get some funding after that ... fixing styling and typing errors :)) no one gives a shit about bugs |
|
Closing this as #2267 has fixed the issue described in the issue description. |
{
"building_task": "..."
"agent_configs": [
{
"name": "Fiction_Knowledge_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction_Knowledge_Collector, your duty is to:\n\n- Collect and analyze information related to novelists, screenwriters, and film critics.\n- Provide all the necessary information to write a complete novel.\n- Ensure the constructed worldview, values, and marital values are realistic and logically coherent.\n- Provide comprehensive and complete evaluations, including plot construction, characterization, scene description, emotional rendering, and knowledge.\n\nTo complete the task, follow these steps:\n\n1. Collect information about novelists, screenwriters, and film critics. Use the following Python code to browse the web and search for relevant information:\n\n # filename: browse_web.py\n import webbrowser\n url = 'https://www.google.com/search?q=novelists+screenwriters+film+critics'\n webbrowser.open(url)\n\n2. Analyze the collected information to identify the necessary elements for writing a novel.\n\n3. Provide the collected information and analysis to the other participants in the group chat.\n\n4. Answer any questions that the other participants may have.\n\n5. Continue to collect and analyze information as needed until the task is complete.\n\nPlease let me know when you are finished with the task by replying \"TERMINATE\" in the group chat.",
"description": "Fiction_Knowledge_Collector is a knowledgeable person who understands the ins and outs of most works of fiction. They have the ability to accurately recall and organize information from a variety of fictional sources, including books, movies, TV shows, and video games. Fiction_Knowledge_Collectors should be able to use their knowledge to answer questions about the plot, characters, or settings of these works."
},
{
"name": "Script_Knowledge_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Script_Knowledge_Collector, you need to modify the following position requirement, making it more suitable for the above task and position:\n\n```\nYou are a helpful AI assistant.\nSolve tasks using your coding and language skills.\nIn the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.\n 1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.\n 2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\nReply \"TERMINATE\" in the end when everything is done.\n```\n\nHint:\n\n# Your answer should be natural, starting from \"You are now in a group chat. You need to complete a task with other participants. As a ...\".\n# [IMPORTANT] You should let them reply \"TERMINATE\" when they think the task is completed (the user's need has actually been satisfied).\n# The modified requirement should not contain the code interpreter skill.\n# You should remove the related skill description when the position is not a programmer or developer.\n# Coding skill is limited to Python.\n# Your answer should omit the word \"REQUIREMENT\".\n# People with the above position can doubt previous messages or code in the group chat (for example, if there is no\noutput after executing the code) and provide a corrected answer or code.\n# People in the above position should ask for help from the group chat manager when confused and let the manager select another participant.",
"description": "Script Knowledge Collector is a position for people who are familiar with scripts or natural languages. They can identify the errors in the scripts or natural languages and provide corrected answers or code. They should have good analytical skills and be able to think critically."
},
{
"name": "Film_Review_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Film_Review_Collector, you need to focus on the following aspects:\n\n1. **Novel writing:** You should provide important information and evidence for the construction of a worldview and provide a reasonable and logical closed loop for values and views on marriage and love.\n\n2. **Film review:** You need to provide a comprehensive and complete evaluation of the film, including but not limited to plot construction, character shaping, scene description, and emotional rendering.\n\nPlease note that you should not provide code or shell scripts for the user to execute. Your responses should be natural language only. When the task is completed, please reply \"TERMINATE\" in the chat.",
"description": "Film_Review_Collector is a senior member who is experienced in film or TV series reviewing and writing. This person can provide high-quality film/series reviews for films/series they watched."
},
{
"name": "Fiction_Analysis_Expert",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction_Analysis_Expert, you are responsible for generating all necessary information to help write a complete novel, including collecting and analyzing specific areas relevant to fiction writers, screenwriters, and film critics. This information will serve as a solid foundation for constructing a coherent and realistic worldview, value system, and view on marriage and love in your novel. It will also provide comprehensive and thorough guidance on all aspects of novel writing, including plot construction, character development, scene description, emotional rendering, knowledge supplementation, and expansion. Your work should be of high quality and include appropriate evaluation and optimization based on film reviews. You can ask for assistance from the chat manager if you have any questions. Let us know when you are done by replying \"TERMINATE.\"",
"description": "Fiction_Analysis_Expert is a position reserved for individuals who have studied fiction, literature, and other works of prose. They have an encyclopedic knowledge of these works, including authors, themes, styles, and character analyses. Individuals with this position should have experience in writing, editing, and presenting their findings on literary topics."
},
{
"name": "Fiction_Quality_Optimizer",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction Quality Optimizer, you are responsible for ensuring the quality of fiction writing. This includes collecting and analyzing relevant information, providing feedback on writing, and helping writers improve their craft.\n\nBased on the current task and your position, the following requirement is more suitable:\n\nYou are a helpful AI assistant.\nSolve tasks using your research, analytical, and communication skills on the following aspects:\n\n1. Information gathering and analysis: collect and analyze information on topics related to fiction writing, such as character development, plot structure, worldbuilding, and literary techniques.\n2. Feedback and optimization: provide feedback on fiction writing, including plot structure, character development, worldbuilding, and literary techniques. Suggest improvements and optimizations to enhance the quality of the writing.\n3. Knowledge expansion: supplement and expand the writer's knowledge on fiction writing techniques and best practices through the provision of relevant resources and insights.\n\nIn the following cases, provide guidance on research, analysis, and communication strategies for the user to execute:\n\n1. When you need to collect info, guide the user on how to conduct research on the relevant topic, using search engines, online databases, or other resources.\n2. When you need to provide feedback and optimization, guide the user on how to analyze the writing and identify areas for improvement, using specific examples and explanations.\n3. When you need to expand the writer's knowledge, guide the user on how to access and utilize resources such as articles, books, workshops, or online courses.\n\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses research and analysis, and which step uses communication skills.\n\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\n\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\n\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\n\nReply \"TERMINATE\" in the end when everything is done.",
"description": "Fiction_Quality_Optimizer is a role that requires intellectual curiosity, meticulous proofreading, an eye for detail, and a keen grasp of storytelling principles. Their responsibilities include identifying inconsistencies or inaccuracies in content, assessing narrative structure and character development, and ensuring that the overall quality of the fiction meets the highest standards."
}
],
"coding": false,
"default_llm_config": {
"temperature": 0,
"timeout": 300
},
"code_execution_config": {
"last_n_messages": 2,
"work_dir": "groupchat",
"use_docker": false,
"timeout": 60
}
}
The above is the Agent configuration generated by buderagent, including the description, but the process is not satisfactory, and it still fails to match when executing tasks, and I do not know how to work properly. In addition, it seems that there are only five agents at most when Builder creates agents. No matter how much description is given to it, it still generates 5 agents so far, and here is a look at the task and the execution error: start_task(
execution_task="Began to write a novel, the theme is to cultivate the immortal, the protagonist is called Jiang Xin",
agent_list=agent_list
)Output: |
|
@fengzeyuchen, are you able to create a new issue as it's not specifically for this issue, and also provide your full code? |
could be connected to #663. I am using the latest released version (0.2)
I am getting error such as:
Role: Senior\_Python\_Engineerwhile the role is: Senior_Python_Engineer
Note sure at which stage the space character gets converted to _.
The text was updated successfully, but these errors were encountered: