[Bug]: Custom Model not supported in Group Chat with Speaker Prompt

[cbrzn]

Describe the bug

When trying to use Autogen only with Custom Models, the functionality of Group Chat with select_speaker_prompt_template attribute doesn't work as expected, because it creates a conversable agent which is not given the custom model client if exists

Steps to reproduce

1- Create GroupChatManager, which receives the LLM config from a local model, and use the register_model_client in the manager
2- Create multiple agents, with the LLM config from a local modal, and use the register_model_client in those
3- Create a Groupchat with select_speaker_prompt_template attribute defined, with the multiple agents
3- Create a user_proxy, which receives the LLM config from a local model and use the register_model_client in the user_proxy
4- Initiate chat between user_proxy and GroupChatManager, when it comes to select an agent it will throw this error because the ConversableAgent that defines the next agent (based on the speaker prompt template) doesn't have the local model

Model Used

https://huggingface.co/meetkai/functionary-small-v2.4-GGUF with LlamaCppPython

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

[Hk669]

@cbrzn, we are enhancing Autogen support for non openai models, feel free to check the roadmap #2946 . thanks

[wireless90]

Hi, i am facing a similar issue.

from typing import Any, List, Dict
from autogen import config_list_from_json, ConversableAgent, AssistantAgent, GroupChat, GroupChatManager

from neuroengine_client import NeuroengineClient
# Import configuration
llm_configuration = config_list_from_json("OAI_CONFIG_LIST.json", filter_dict={"model_client_cls": ["NeuroengineClient"]})
llm_configuration = {"config_list": llm_configuration}

# System Messages
ADDER_SYS_MESSAGE = "You will be given an integer, simply add one to it and return the result"
SUBTRACTOR_SYS_MESSAGE = "You will be given an integer, simply subtract one to it and return the result"
MULTIPLIER_SYS_MESSAGE = "You will be given an integer, simply multiply by 3 and return the result"
DIVIDER_SYS_MESSAGE = "You will be given an integer, simply DIVIDE by 3 and round down to the nearest integer, and return the result"
MASTER_SYS_MESSAGE = "You will be given a starting number and a target number. Use the other agents to turn the starting number into the target number. Then terminate group chat."
NUMBER_SYS_MESSAGE = "Simply return back the number."

# Descriptions
ADDER_DESCRIPTION = "Simply adds 1 to a given number."
SUBTRACTOR_DESCRIPTION = "Simply subtracts 1 to a given number."
MULTIPLIER_DESCRIPTION = "Simply multiples 3 to a given number."
DIVIDER_DESCRIPTION = "Simply divides 3 to a given number and rounds down to nearest integer."
NUMBER_DESCRIPTION = "Simply returns back the given number."

# Create agents
adder_agent = AssistantAgent(name="adder_agent", system_message=ADDER_SYS_MESSAGE, llm_config=llm_configuration, description=ADDER_DESCRIPTION)
subtractor_agent = AssistantAgent(name="subtractor_agent", system_message=SUBTRACTOR_SYS_MESSAGE, llm_config=llm_configuration, description=SUBTRACTOR_DESCRIPTION)
multiplier_agent = AssistantAgent(name="multiplier_agent", system_message=MULTIPLIER_SYS_MESSAGE, llm_config=llm_configuration, description=MULTIPLIER_DESCRIPTION)
divider_agent = AssistantAgent(name="divider_agent", system_message=DIVIDER_SYS_MESSAGE, llm_config=llm_configuration, description=DIVIDER_DESCRIPTION)
number_agent = AssistantAgent(name="number_agent", system_message=NUMBER_SYS_MESSAGE, llm_config=llm_configuration, description=NUMBER_DESCRIPTION)

# Register model client for each agent
for agent in [adder_agent, subtractor_agent, multiplier_agent, divider_agent, number_agent]:
    agent.register_model_client(model_client_cls=NeuroengineClient)

# Create group chat and group chat manager
group_chat = GroupChat(agents=[adder_agent, subtractor_agent, multiplier_agent, divider_agent, number_agent], messages=[])
group_chat_manager = GroupChatManager(name="master_agent", system_message=MASTER_SYS_MESSAGE, llm_config=llm_configuration, groupchat=group_chat)
group_chat_manager.register_model_client(model_client_cls=NeuroengineClient)

# Initiate chat
number_agent.initiate_chat(group_chat_manager, message="My number is 3, I want to turn it into 13.", summary_method="reflection_with_llm")

[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
number_agent (to master_agent):

My number is 3, I want to turn it into 13.

--------------------------------------------------------------------------------
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
Traceback (most recent call last):
  File "c:/Users/wirel/OneDrive/Documents/Python/neuro_test/main_groupchat.py", line 41, in <module>
    number_agent.initiate_chat(group_chat_manager, message="My number is 3, I want to turn it into 13.", summary_method="reflection_with_llm")
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1018, in initiate_chat
    self.send(msg2send, recipient, silent=silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 818, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1972, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 1047, in run_chat 
    speaker = groupchat.select_speaker(speaker, self)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 538, in select_speaker
    return self._auto_select_speaker(last_speaker, selector, messages, agents)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 658, in _auto_select_speaker
    result = checking_agent.initiate_chat(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1011, in initiate_chat
    self.send(msg2send, recipient, request_reply=True, silent=silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 818, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1972, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1340, in generate_oai_reply
    extracted_response = self._generate_oai_reply_from_client(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1359, in _generate_oai_reply_from_client
    response = llm_client.create(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\oai\client.py", line 639, in create
    raise RuntimeError(
RuntimeError: Model client(s) ['NeuroengineClient'] are not activated. Please register the custom model clients using `register_model_client` or filter them out form the config list.

[Hk669]

@wireless90 sorry for the issue, but we dont support NeuroengineClient, please do check out the blog on how to register a custom model client at https://microsoft.github.io/autogen/blog/2024/01/26/Custom-Models/

[wireless90]

@wireless90 sorry for the issue, but we dont support NeuroengineClient, please do check out the blog on how to register a custom model client at https://microsoft.github.io/autogen/blog/2024/01/26/Custom-Models/

Hi, yes im using the custom model client. Agent to agent chat it works. Group chat its failing as mentioned above.

Downgrading works, as mentioned here.

#2929 (comment)

# NeuroengineClient.py
from typing import Dict, Any, List
from neuroengine import Neuroengine
from openai.types.chat import ChatCompletion
from openai.types.completion_usage import CompletionUsage
from openai.types.chat.chat_completion import ChatCompletionMessage, Choice
from types import SimpleNamespace
import time

class NeuroengineClient:
    def __init__(self, config: Dict[str, Any], **kwargs):
        self.neuroengine = Neuroengine(
            service_name=config.get("service_name"),
            server_address=config.get("server_address"),
            server_port=config.get("server_port", 443),
            key=config.get("key", ""),
            verify_ssl=config.get("verify_ssl", True)
        )
        self.model_name = config["model"]
        self.id = 1

    def create(self, params: Dict[str, Any]) -> ChatCompletion:
        #print("Create method called with params:", params)  # Debug statement
        messages = params["messages"]
        
        # Convert the list of messages into a single prompt
        # TODO See if needs a prompt summary
        prompt = "\n".join([f'{msg["role"]}:{msg["content"]}' for msg in messages])
        
        response = self.neuroengine.request(prompt, max_new_len=4000)
        
        prompt_tokens = len(prompt.split())
        completion_tokens = len(response.split())
        total_tokens = prompt_tokens + completion_tokens

        message = ChatCompletionMessage(
            role="assistant",
            content=response,
            function_call=None,
            tool_calls=None
        )
        choices = [Choice(finish_reason="stop", index=0, message=message)]

        usage = SimpleNamespace(
            prompt_tokens=prompt_tokens,
            completion_tokens=completion_tokens,
            total_tokens=total_tokens
        )

        chat_completion = ChatCompletion(
            id=str(self.id),  # Generate or fetch a unique ID as per your requirements
            created=int(time.time()),
            object='chat.completion',
            model=self.model_name,
            choices=choices,
            #usage=usage,
            cost=0.0  # Adjust the cost calculation as needed
        )
        self.id += 1
        return chat_completion

    def message_retrieval(self, response: ChatCompletion) -> List[str]:
        return [choice.message.content for choice in response.choices]

    def cost(self, response: ChatCompletion) -> float:
        return 0.0

    @staticmethod
    def get_usage(response: ChatCompletion) -> Dict[str, Any]:
        return dict()
        ```

[kaustubhrm]

any fixes on this issue yet?

[bird-cat]

It seems the bug is not fixed yet in v0.2.38.

[ekzhu]

@bird-cat have you tried setting select_speaker_auto_llm_config?

https://microsoft.github.io/autogen/0.2/docs/reference/agentchat/groupchat

[bird-cat]

@ekzhu setting select_speaker_auto_llm_config works, thank you.
Besides, the user guide https://microsoft.github.io/autogen/0.2/docs/topics/groupchat/using_custom_models also provides a solution.

[Rushikesh67]

@bird-cat, I tried the solution mentioned in the user guide https://microsoft.github.io/autogen/0.2/docs/topics/groupchat/using_custom_models, but i am getting a error as TypeError: GroupChat.init() got an unexpected keyword argument 'model_client_cls'.

[ekzhu]

@bird-cat, I tried the solution mentioned in the user guide https://microsoft.github.io/autogen/0.2/docs/topics/groupchat/using_custom_models, but i am getting a error as TypeError: GroupChat.init() got an unexpected keyword argument 'model_client_cls'.

What's your package version

[Rushikesh67]

@ekzhu pyautogen v0.4

[ekzhu]

@ekzhu pyautogen v0.4

Uninstall this and install autogen-agentchat. We don't have access to pyautogen at the moment.

[Rushikesh67]

@ekzhu still getting the same error, setting speaker_selection_method = "random" is working but https://microsoft.github.io/autogen/0.2/docs/topics/groupchat/using_custom_models is not working
my code-

import os
import requests
from types import SimpleNamespace
from autogen import AssistantAgent, UserProxyAgent, GroupChat, GroupChatManager
import getpass

api_key = getpass.getpass("Please enter your API key: ").strip()

gpt_config = {
    "api_key": api_key,
    "base_url": "https://xyz.com",
    "model": " llama",
    "params": {
        "max_length": 1024,
        "temperature": 0.3,
    },
}

class CustomModelClientForAPI:
    def __init__(self, config):
        self.api_key = config["api_key"]
        self.base_url = config["base_url"]
        self.model = config["model"]
        self.max_length = config["params"]["max_length"]
        self.temperature = config["params"]["temperature"]

    def create(self, params):
        headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json",
        }

        payload = {
            "model": self.model,
            "messages": params["messages"],
            "max_tokens": self.max_length,
            "temperature": self.temperature,
        }

        response = requests.post(
            f"{self.base_url}/chat/completions", headers=headers, json=payload
        )

        if response.status_code == 200:
            data = response.json()
            return SimpleNamespace(
                choices=[
                    SimpleNamespace(
                        message=SimpleNamespace(content=choice["message"]["content"])
                    )
                    for choice in data.get("choices", [])
                ],
                model=data.get("model", self.model),
            )
        else:
            raise Exception(f"Error {response.status_code}: {response.text}")

    def message_retrieval(self, response):
        return [choice.message.content for choice in response.choices]

    @staticmethod
    def cost(response):
        response.cost = 0
        return 0

    @staticmethod
    def get_usage(response):
        return {}

config_list_custom = [
    {
        "model": gpt_config["model"],
        "model_client_cls": "CustomModelClientForAPI",
        "api_key": gpt_config["api_key"],
        "base_url": gpt_config["base_url"],
        "params": gpt_config["params"],
    }
]



agent1 = AssistantAgent("agent1", llm_config={"config_list": config_list_custom}, system_message='''.''',)
engineer.register_model_client(CustomModelClientForAPI)

agent2= AssistantAgent("agent2", llm_config={"config_list": config_list_custom}, system_message='''.''',)
scientist.register_model_client(CustomModelClientForAPI)

agent3= AssistantAgent("agent3", llm_config={"config_list": config_list_custom}, system_message='''.''',)
planner.register_model_client(CustomModelClientForAPI)

agent4= AssistantAgent("agent4", llm_config={"config_list": config_list_custom}, system_message='''.''',)
critic.register_model_client(CustomModelClientForAPI)

user_proxy = UserProxyAgent(
    "Admin",
    human_input_mode="NEVER",
    system_message=".",
    code_execution_config={"work_dir": "coding", "use_docker": False},
)

executor= UserProxyAgent(
    "executor",
    system_message=".",
    human_input_mode="NEVER",
    code_execution_config={"work_dir": "coding", "use_docker": False},
)


groupchat = GroupChat(agents=[user_proxy, agent1, agent2, agent3, agent4, executor], messages=[], model_client_cls=CustomModelClientForAPI , max_round=50)
manager = GroupChatManager(groupchat=groupchat, llm_config={"config_list": config_list_custom})
manager.register_model_client(CustomModelClientForAPI)

# Define task message
task_message = """  .  """

try:
    user_proxy.initiate_chat(manager, message=task_message)
except Exception as e:
    print(f"Error during chat initiation: {e}")

[ekzhu]

@Rushikesh67 just checked, the argument should be select_speaker_auto_model_client_cls

https://github.com/microsoft/autogen/blob/ebb3e24c51e0bfc3083065ce116918118480d437/autogen/agentchat/groupchat.py#L147C5-L147C41

[Rushikesh67]

@ekzhu I modified it to

groupchat = GroupChat(agents=[user_proxy, planner, engineer, scientist, executor, critic], messages=[],select_speaker_auto_verbose=True,
    select_speaker_auto_llm_config=config_list_custom,
    select_speaker_auto_model_client_cls=CustomModelClientForAPI,
    max_round=20)
manager = GroupChatManager(groupchat=groupchat, llm_config={"config_list": config_list_custom})
manager.register_model_client(CustomModelClientForAPI)

but now I am getting

Error during chat initiation: llm_config must be a dict or False or None.

How am I supposed to set llm_config to False or None when I have a custom model, llm_config={"config_list": config_list_custom}

[ekzhu]

What's in the config_list_custom

[Rushikesh67]

@ekzhu

config_list_custom = [
    {
        "model": gpt_config["model"],
        "model_client_cls": "CustomModelClientForAPI",
        "api_key": gpt_config["api_key"],
        "base_url": gpt_config["base_url"],
        "params": gpt_config["params"],
    }
]

this was the full code for your reference #2956 (comment)