Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: generate_init_message fails for multimodal model. #1924

Closed
BeibinLi opened this issue Mar 8, 2024 · 0 comments · Fixed by #2124
Closed

[Bug]: generate_init_message fails for multimodal model. #1924

BeibinLi opened this issue Mar 8, 2024 · 0 comments · Fixed by #2124
Assignees

Comments

@BeibinLi
Copy link
Collaborator

BeibinLi commented Mar 8, 2024

Describe the bug

Note that the second method (in the bottom) does not work today, because of recent updates in generate_init_message

image_agent = MultimodalConversableAgent(
    name="image-explainer",
    max_consecutive_auto_reply=10,
    llm_config={"config_list": config_list_4v, "temperature": 0.5, "max_tokens": 300},
)

user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    system_message="A human admin.",
    human_input_mode="NEVER",  # Try between ALWAYS or NEVER
    max_consecutive_auto_reply=0,
    code_execution_config={
        "use_docker": False
    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.
)

# Ask the question with an image
user_proxy.initiate_chat(
    image_agent,
    message="""What's the breed of this dog?
<img https://th.bing.com/th/id/R.422068ce8af4e15b0634fe2540adea7a?rik=y4OcXBE%2fqutDOw&pid=ImgRaw&r=0>.""",
)

# Equivalent
image_agent.clear_history()
user_proxy.initiate_chat(
    image_agent,
    message=[
        {
        "type": "text",
        "text": "What's the breed of this dog?"
        },
        {
        "type": "image_url",
        "image_url": {
            "url": get_pil_image("https://th.bing.com/th/id/R.422068ce8af4e15b0634fe2540adea7a?rik=y4OcXBE%2fqutDOw&pid=ImgRaw&r=0")
                },
        },
        {
        "type": "text",
        "text": "."
        },
    ]
)

Error message:

File ~/autogen/autogen/agentchat/conversable_agent.py:622, in ConversableAgent.send(self, message, recipient, request_reply, silent)
    620     recipient.receive(message, self, request_reply, silent)
    621 else:
--> 622     raise ValueError(
    623         "Message can't be converted into a valid ChatCompletion message. Either content or function_call must be provided."
    624     )

ValueError: Message can't be converted into a valid ChatCompletion message. Either content or function_call must be provided.

Steps to reproduce

No response

Model Used

No response

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants