When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors #4422

gianx89 · 2024-11-28T16:17:33Z

Hi to all! I'm using Autogen to develop a RAG system, even better if Multi-Agent. I must use open source, preferably local models, like llama 3.1 or llama 3.2 . I'm using ChromaDB as my vector database.

I'm developing a system that can write a Comic Book Story with a specific format. There is writer or teams of writers that writes the story, a critic or a team of critics that gives advices on how to improve the story a manager or team of managers that incorporates this suggestions. When the story is deemed satisfactory, an agent writes "TERMINATE".

I don't have any issues using OpenAI APIs and models like GPT-3.5 Turbo or GPT-4. However, when working with open-source or local models, I encounter unpredictable behaviors.

The agents starts talking:

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-11-28 16:51:13,962 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-11-28 16:51:14,089 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel comics, to manga and graphic novels, all play a part in enriching the minds of the young and the young-at-heart.


...


--------------------------------------------------------------------------------

Next speaker: S0

Truncated 2957 tokens. Number of tokens reduced from 3366 to 409
S0 (to chat_manager):

TITLE: The Art of Superhero Storytelling
ABSTRACT: A young writer learns the art of crafting superhero comic book stories from an experienced professor.

PROTAGONISTS: 
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex: A new writer who is learning to create superhero comic book stories.

PANEL START 1
 IMAGE_DESCRIPTION: Professor Thompson's office with shelves of comic books and a desk with writing materials.
 IMAGE_DIALOGUES: 
Professor Thompson: "Welcome, Alex. I see you're interested in writing superhero comics."
Alex: "Yes, Professor. I've always been a fan."
PANEL END 1

PANEL START 2
 IMAGE_DESCRIPTION: Professor Thompson holds up a comic book.
 IMAGE_DIALOGUES: 
Professor Thompson: "The key to a great superhero story is a compelling character arc."
PANEL END 2

PANEL START 3
 IMAGE_DESCRIPTION: Alex scribbles notes as Professor Thompson lectures.
 IMAGE_DIALOGUES: 
Professor Thompson: "You must balance action and dialogue to keep the reader engaged."
Alex: "That makes sense, but how do I do that?"
PANEL END 3

PANEL START 4
 IMAGE_DESCRIPTION: Professor Thompson shows Alex a split-screen of a comic panel.
 IMAGE_DIALOGUES: 
Professor Thompson: "Panel layout is crucial. It guides the reader's eye through the story."
PANEL END 4

PANEL START 5
 IMAGE_DESCRIPTION: Alex and Professor Thompson sit at a table with a large sheet of paper and pencils.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's create a story together. What's your superhero's origin story?"
Alex: "Uh, I'm not sure..."
PANEL END 5

PANEL START 6
 IMAGE_DESCRIPTION: Professor Thompson smiles as Alex starts to brainstorm.
 IMAGE_DIALOGUES: 
Professor Thompson: "That's it! The origin story is the heart of any superhero tale."
PANEL END 6

PANEL START 7
 IMAGE_DESCRIPTION: Alex and Professor Thompson work on the comic script together.
 IMAGE_DIALOGUES: 
Alex: "I think I'm getting the hang of this!"
Professor Thompson: "You are! Just remember to keep it concise."
PANEL END 7

PANEL START 8
 IMAGE_DESCRIPTION: Alex and Professor Thompson review the comic script.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's add some action scenes to keep the reader engaged."
Alex: "I've got an idea!"
PANEL END 8

PANEL START 9
 IMAGE_DESCRIPTION: Alex and Professor Thompson work on the comic's action scenes.
 IMAGE_DIALOGUES: 
Alex: "This is so much fun! I never realized how much work went into a comic."
Professor Thompson: "That's the beauty of it – collaboration and creativity."
PANEL END 9

PANEL START 10
 IMAGE_DESCRIPTION: Alex holds up the finished comic book, beaming with pride.
 IMAGE_DIALOGUES: 
Alex: "Thank you, Professor Thompson. I couldn't have done it without you."
Professor Thompson: "You're welcome, Alex. You're a natural."
PANEL END 10

NEXT: C0

--------------------------------------------------------------------------------

Next speaker: C0

Truncated 3204 tokens. Number of tokens reduced from 4022 to 818
C0 (to chat_manager):

NEXT: C1

--------------------------------------------------------------------------------

Next speaker: C1

Truncated 3204 tokens. Number of tokens reduced from 4026 to 822
C1 (to chat_manager):

The story is off to a good start. I like how Professor Thompson is guiding Alex through the basics of superhero storytelling.

However, I think the pacing could be improved. Some panels feel a bit rushed, such as panel 5 where Alex is asked to come up with an origin story on the spot.

I'd like to see more development of the mentor-student relationship and more exploration of the creative process.

NEXT: C0

--------------------------------------------------------------------------------

Next speaker: C0

Truncated 3204 tokens. Number of tokens reduced from 4112 to 908
C0 (to chat_manager):

I agree with C1's assessment. The story has potential, but the pacing and character development need improvement. I suggest adding more panels to show the progression of Alex's learning and the mentor-student relationship.

Specifically, I'd like to see more of Professor Thompson's teaching style and how Alex responds to it. Perhaps we could add a panel where Alex shares their own ideas and Professor Thompson provides feedback.

Additionally, let's explore the creative process more deeply. What are the challenges of writing a superhero story? How does Professor Thompson help Alex overcome these challenges?

NEXT: S0

--------------------------------------------------------------------------------

Next speaker: S0

Truncated 3204 tokens. Number of tokens reduced from 4229 to 1025
S0 (to chat_manager):

TITLE: The Art of Superhero Storytelling
ABSTRACT: A young writer learns the art of crafting superhero comic book stories from an experienced professor.

PROTAGONISTS: 
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex: A new writer who is learning to create superhero comic book stories.

PANEL START 1
 IMAGE_DESCRIPTION: Professor Thompson's office with shelves of comic books and a desk with writing materials.
 IMAGE_DIALOGUES: 
Professor Thompson: "Welcome, Alex. I see you're interested in writing superhero comics."
Alex: "Yes, Professor. I've always been a fan."
PANEL END 1

PANEL START 2
 IMAGE_DESCRIPTION: Professor Thompson holds up a comic book.
 IMAGE_DIALOGUES: 
Professor Thompson: "The key to a great superhero story is a compelling character arc."
PANEL END 2

PANEL START 3
 IMAGE_DESCRIPTION: Alex scribbles notes as Professor Thompson lectures.
 IMAGE_DIALOGUES: 
Professor Thompson: "You must balance action and dialogue to keep the reader engaged."
Alex: "That makes sense, but how do I do that?"
PANEL END 3

PANEL START 4
 IMAGE_DESCRIPTION: Professor Thompson shows Alex a split-screen of a comic panel.
 IMAGE_DIALOGUES: 
Professor Thompson: "Panel layout is crucial. It guides the reader's eye through the story."
PANEL END 4

PANEL START 5
 IMAGE_DESCRIPTION: Alex and Professor Thompson sit at a table with a large sheet of paper and pencils.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's brainstorm together. What kind of superhero would you like to create?"
Alex: "I've always wanted to write about a hero with the power of teleportation."
PANEL END 5

PANEL START 6
 IMAGE_DESCRIPTION: Professor Thompson smiles as Alex starts to brainstorm.
 IMAGE_DIALOGUES: 
Professor Thompson: "That's a great start! But how do you plan to make this hero relatable to readers?"
Alex: "Hmm, I'm not sure..."
PANEL END 6

PANEL START 7
 IMAGE_DESCRIPTION: Professor Thompson provides feedback as Alex shares their ideas.
 IMAGE_DIALOGUES: 
Professor Thompson: "I like where you're going with this, but what if we add a personal stake to the hero's teleportation power?"
Alex: "Like what?"
PANEL END 7

PANEL START 8
 IMAGE_DESCRIPTION: Alex's eyes light up as they start to see the potential of their hero.
 IMAGE_DIALOGUES: 
Professor Thompson: "Exactly! Now we're getting somewhere. The hero's power is not just a tool, but a reflection of their inner struggle."
Alex: "That's amazing!"
PANEL END 8

PANEL START 9
 IMAGE_DESCRIPTION: Professor Thompson hands Alex a script with notes and feedback.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now it's your turn to write a scene. Don't worry about getting it perfect, just let your creativity flow."
Alex: "Thanks, Professor. I'll get right on it!"
PANEL END 9

PANEL START 10
 IMAGE_DESCRIPTION: Alex sits at their desk, writing furiously as Professor Thompson looks on with a proud smile.
 IMAGE_DIALOGUES: 
Alex: "This is it! I think I'm finally getting the hang of it!"
Professor Thompson: "That's the spirit, Alex! You're well on your way to becoming a great superhero writer."
PANEL END 10

NEXT: P0

--------------------------------------------------------------------------------

Next speaker: P0

Truncated 3520 tokens. Number of tokens reduced from 4954 to 1434
==============================
Chat Ends
==============================

Process finished with exit code 0

The chats ends without error but abruptly, without meeting the termination condition. Sometimes, very rarely, I get the right results and the chat ends with "TERMINATE" string.

Any suggestions? I can't share all the code at the moment, but I can reply with snippets of it.

P.S.
The max_rounds are pretty high (100)

The text was updated successfully, but these errors were encountered:

MohMaz · 2024-11-28T23:13:30Z

@gianx89 thanks for raising the issue. Will you be able to share more how you are provisioning your local API? Anything that might help reproducing?

gianx89 · 2024-11-29T00:28:52Z

Hi and thanks, @MohMaz.

I’ve tried the following provisioning modes:

Ollama (local)
Ollama with liteLLM (local)
together.ai (remote)

I’ve been focusing on various Llama versions and tested them all in different "sizes". Here’s what I’ve tried:

meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
llama3.1
llama3.2
llama3.2-vision

I’ve also experimented with Mistral and other models in the RAG-only version of my project, but I consistently encountered problems. Now, I’m shifting my focus to the Multi-Agent aspect of the project.

I’ve tried reducing the token size (using an approximate method) and limiting the size of the chat history, but the results remain the same.

Here’s my poetry.toml file (I’m experimenting with several libraries in other packages):

[tool.poetry]
name = "comics"
version = "0.0.1"
description = "Multi Agent RAG System to write a comic book story"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11,<3.13"
atomic-agents = "^1.0.10"
autogen = "^0.3.1"
deepeval = "^1.5.0"
instructor = "^1.0.1"
rich = "^13.9.4"
instructorembedding = "^1.0.1"
sentence-transformers = "^3.3.0"
emoji = "^2.14.0"
ipython = "^8.29.0"
markdownify = "^0.13.1"
flaml = {extras = ["automl"], version = "^2.3.2"}
pydantic = "2.9.2"
pydantic-core = "2.23.4"
ollama = "^0.3.3"
httpx = "^0.27.2"
ollama-api = "^0.1.1"
autogen-agentchat = {extras = ["together"], version = "^0.2.39"}
matplotlib = "^3.9.2"
groq = "^0.12.0"
llama-index = "^0.12.1"
llama-index-llms-ollama = "^0.4.1"
chromadb = "^0.5.20"
llama-index-vector-stores-chroma = "^0.4.0"
llama-index-embeddings-ollama = "^0.4.0"
llama-index-embeddings-huggingface = "^0.4.0"
langgraph = "^0.2.53"
pyautogen = "^0.4"
litellm = {extras = ["proxy"], version = "^1.52.16"}
together = "^1.3.5"
langchain-together = "^0.2.0"
tiktoken = "^0.8.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Reproducing the Problem

This is a link to a Python file to reproduce the problem.

Currently, it uses meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo from together.ai, but you can comment out the code to switch to using the local llama3.2-vision via Ollama.

You must provide some documents in the RAG documents folder; otherwise, you might not be able to reproduce the issue. The problem sometimes worsens when I include context from RAG.

https://www.dropbox.com/scl/fi/oslxnk78soz208wrn2gyc/autogen_multiagent_rag.py?rlkey=67u8017sn9r3fi0wat764rrsh&st=5t9lp5rm&dl=0

ekzhu · 2024-11-29T19:39:21Z

I think the termination condition was triggered. It could be the message transform that truncates the message and prevents the message from displaying fully. @thinkall @WaelKarkoub what do you think?

gianx89 · 2024-11-29T20:30:27Z

It happened even without truncation. I added it trying to solve the problem. I'll post later an output without truncation. Or you can try it disabling the truncation in the provided script.

gianx89 · 2024-11-29T20:52:01Z

Here it's an output without truncation, the same problem happens.

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-11-29 21:45:19,089 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-11-29 21:45:19,243 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel 

...

NEXT: S0

--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

I'll start creating the story based on your idea. Here's the first draft:

TITLE: The Art of Superhero Storytelling
ABSTRACT: A seasoned comic book writer and professor teaches a new writer the art of crafting superhero comic book stories.

PROTAGONISTS:
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex Chen: A new writer eager to learn the art of creating superhero comic book stories.

PANEL START 1
  IMAGE_DESCRIPTION: Professor Thompson sitting at a desk with various comic books and papers scattered around. Alex Chen entering the room, looking a bit nervous.
  IMAGE_DIALOGUES:
    Professor Thompson: "Welcome to the world of superhero comics! I'm excited to share my knowledge with you."
    Alex Chen: "I've always been a huge fan of comics, but I have no idea where to start."
PANEL END 1

PANEL START 2
  IMAGE_DESCRIPTION: Close-up of Professor Thompson's face, smiling and enthusiastic.
  IMAGE_DIALOGUES:
    Professor Thompson: "Don't worry, Alex. We'll start with the basics. What makes a great superhero story?"
PANEL END 2

PANEL START 3
  IMAGE_DESCRIPTION: Alex Chen looking thoughtful, with a speech bubble above their head.
  IMAGE_DIALOGUES:
    Alex Chen: "I think it's the hero's journey. The struggle between good and evil."
PANEL END 3

PANEL START 4
  IMAGE_DESCRIPTION: Professor Thompson nodding in agreement, with a comic book panel in the background showing a classic superhero battle scene.
  IMAGE_DIALOGUES:
    Professor Thompson: "That's right! The hero's journey is a classic trope in superhero comics. But what about the characters themselves? What makes them relatable and engaging?"
PANEL END 4

PANEL START 5
  IMAGE_DESCRIPTION: Alex Chen looking puzzled, with a thought bubble above their head.
  IMAGE_DIALOGUES:
    Alex Chen: "I'm not sure. I've always just focused on the action and adventure."
PANEL END 5

PANEL START 6
  IMAGE_DESCRIPTION: Professor Thompson smiling, with a whiteboard behind them filled with notes and diagrams.
  IMAGE_DIALOGUES:
    Professor Thompson: "Ah, that's where character development comes in. Let me show you some techniques for creating well-rounded characters."
PANEL END 6

PANEL START 7
  IMAGE_DESCRIPTION: Professor Thompson drawing on the whiteboard, with Alex

--------------------------------------------------------------------------------
==============================
Chat Ends
==============================

Process finished with exit code 0

This is the link containing the code to reproduce the error.

https://www.dropbox.com/scl/fi/k27e6xp3tdp75bbefisoo/autogen_multiagent_rag_no_trunc.py?rlkey=huu8a1vc3r08h7yvmhme09l98&dl=0

thinkall · 2024-12-03T09:11:26Z

Hi @gianx89 , the chat is terminated by the RAG agent. To fix it, you'll need to pass in your own "is_termination_msg" to it like you've done for GroupChatManager.

You may want to check out https://microsoft.github.io/autogen/0.2/docs/topics/retrieval_augmentation , https://github.com/microsoft/autogen/blob/0.2/notebook/agentchat_groupchat_RAG.ipynb and https://github.com/microsoft/autogen/blob/0.2/notebook/agentchat_RetrieveChat.ipynb for some examples of RAG agents.

You've mentioned that with openai models, everything is fine. I'm curious, because with your code, I'd expect the same termination behavior.

gianx89 · 2024-12-03T17:24:35Z

@thinkall Hi and thanks. I didn't try this exact code iteration with OpenAI. In general, the same code performed better and in a most reliable way with OpenAI APIs.

I tried adding is_termination_msg=is_termination_msg to all agents and I put a breakpoint on the return true, never enters and the chat ends anyway:

def is_termination_msg(content) -> bool:
    have_content = content.get("content", None) is not None
    if have_content and "TERMINATE" in content["content"]:
        return True
    return False

I tried adding is_termination_msg=is_termination_msg only to the RAG agent, the result is the same. Later I'll try again with OpenAI APis.

This is an example output:

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-12-03 18:20:09,318 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-12-03 18:20:09,440 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel 

...

--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

NEXT: S1.

--------------------------------------------------------------------------------

Next speaker: S1

S1 (to chat_manager):

I can assist with brainstorming the story for our comic, but I think S0 should lead the discussion on the story and its creation.

However, I can start by suggesting some possible plot points and character developments that might fit the guidelines. For example, we could start with an intro panel showing the professor's lecture hall, with the professor teaching a class on superhero comic book creation. The new writer could be shown sitting in the class, looking enthusiastic but also a bit lost.

In the next few panels, we could see the professor and the new writer having a one-on-one meeting to discuss their project. The professor could be shown providing guidance and feedback on the new writer's ideas, while the new writer could be shown struggling to come up with a compelling story.

We could also include some panels showing the new writer's creative process, such as brainstorming ideas, researching different superheroes, and sketching out rough drafts of their story. The professor could be shown offering words of encouragement and advice throughout this process.

As the story progresses, we could see the new writer's skills and confidence growing, with the professor providing support and guidance every step of the way. We could also include some panels showing the finished comic book, with the new writer's artwork and storytelling bringing the professor's teachings to life.

I'll stop here for now and let S0 take over. NEXT: S0

--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

Thank you, S1, for your suggestions. I'll build upon them to create a first draft of the story.

Here is a possible comic book story that meets the guidelines:

**TITLE:** "The Art of Superhero Storytelling"

**ABSTRACT:** A new writer learns the art of creating superhero comic books from an experienced professor, navigating the challenges of storytelling, character development, and visual creativity.

**PROTAGONISTS:**

* Professor Thompson: An experienced superhero comic writer and professor who teaches a class on comic book creation.
* Alex Chen: A new writer who is eager to learn the art of creating superhero comic books.

**PANEL START 1**
IMAGE_DESCRIPTION: A lecture hall with Professor Thompson standing at the front, writing on the blackboard. Alex Chen sits in the audience, looking enthusiastic but also a bit lost.
IMAGE_DIALOGUES:
Professor Thompson: "Welcome, students, to our class on superhero comic book creation. Today, we'll be discussing the importance of storytelling in comics."

**PANEL START 2**
IMAGE_DESCRIPTION: Professor Thompson and Alex Chen meet one-on-one in the professor's office. Professor Thompson is reviewing Alex's script, while Alex looks anxious.
IMAGE_DIALOGUES:
Professor Thompson: "Alex, your ideas are great, but we need to work on developing your characters. Can you tell me more about your hero's backstory?"

**PANEL START 3**
IMAGE_DESCRIPTION: Alex is shown brainstorming ideas, researching different superheroes, and sketching out rough drafts of their story.
IMAGE_DIALOGUES:
Alex (thought bubble): "I want my hero to be different... but also relatable. What kind of powers should they have?"

**PANEL START 4**
IMAGE_DESCRIPTION: Professor Thompson is shown offering words of encouragement and advice to Alex.
IMAGE_DIALOGUES:
Professor Thompson: "Remember, Alex, the key to creating a great superhero comic is to balance action, drama, and humor. Don't be afraid to take risks and try new things."

**PANEL START 5**
IMAGE_DESCRIPTION: Alex's artwork and storytelling are shown, bringing the professor's teachings to life. The panel shows a dynamic battle scene between the hero and a villain.
IMAGE_DIALOGUES:
Alex (captions): "In a world where superpowers are the norm, one hero dares to be different..."

**PANEL START 6**
IMAGE_DESCRIPTION: The new writer's creative process is shown, with rough drafts, revisions, and feedback from the professor.
IMAGE_DIALOGUES:
Professor Thompson:

--------------------------------------------------------------------------------
==============================
Chat Ends
==============================

Process finished with exit code 0

This is the revised code:

https://www.dropbox.com/scl/fi/982ujaaey7dlsv8bvktde/autogen_multiagent_rag_is_termination_msg.py?rlkey=fng5igi7zqguckbhddu7ub4ku&dl=0

Any suggestions?

thinkall · 2024-12-04T06:25:03Z

Hi @gianx89 , as I tried your code, the chat terminated because there is NoEligibleSpeaker. You can update your custom_speaker_selection_func to ensure a default next_speaker for each current_speaker. I've also raised a PR #4535 to increase the readability of the outputs.

gianx89 · 2024-12-04T18:02:42Z

@thinkall I tried to fix the custom_speaker_selection_func function like this:

def custom_speaker_selection_func(last_speaker: Agent, groupchat: GroupChat):
    """Define a customized speaker selection function.
    A recommended way is to define a transition for each speaker in the groupchat.

    Returns:
        Return an Agent class or a string from ['auto', 'manual', 'random', 'round_robin'] to select a default method to use.
    """
    messages = groupchat.messages

    # print (messages[-1])

    if len(messages) <= 1:
        # First message goes to S0
        return agents[3]

    if last_speaker is agents[0]:  # "P0"
        if "NEXT: S" in messages[-1]["content"]:
            # If the last message has redirect to Story group, let S0 to speak
            return agents[3]  # "S0"
        elif "NEXT: C" in messages[-1]["content"]:
            # If the last message has redirect to Critic group, let C0 to speak
            return agents[6]  # "C0"
        elif "NEXT: P1" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[1]  # "P1"
        elif "NEXT: P2" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[2]  # "P2"
        else:
            # if nothing matches, always let P0 to speak
            return agents[6]
            # return "random"

    elif last_speaker is agents[1]:  # P1
        if "NEXT: P2" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[2]
        else:
            # All other cases redirect P1 and P2 messages to P0
            return agents[0]
    elif last_speaker is agents[2]:  # P2
        if "NEXT: P1" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[1]
        else:
            # All other cases redirect P1 and P2 messages to P0
            return agents[0]

    elif last_speaker is agents[3]:  # "S0"
        if "NEXT: P" in messages[-1]["content"]:
            # If the last message has redirect to Primary group, let P0 to speak
            return agents[0]  # "P0"
        elif "NEXT: C" in messages[-1]["content"]:
            # If the last message has redirect to Critic group, let C0 to speak
            return agents[6]  # "C0"
        elif "NEXT: S1" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[4]  # "S1"
        elif "NEXT: S2" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[5]  # "S2"
        else:
            # All other cases redirect P1 and P2 messages to P0
            return agents[0]

    elif last_speaker is agents[4]:  # S1
        if "NEXT: S2" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[5]
        else:
            # All other cases redirect S1 and S2 messages to S0
            return agents[3]
    elif last_speaker is agents[5]:  # S2
        if "NEXT: S1" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[4]
        else:
            # All other cases redirect S1 and S2 messages to S0
            return agents[3]

    elif last_speaker is agents[6]:  # "C0"
        if "NEXT: P" in messages[-1]["content"]:
            # If the last message has redirect to Primary group, let P0 to speak
            return agents[0]  # "P0"
        elif "NEXT: S" in messages[-1]["content"]:
            # If the last message has redirect to Story group, let S0 to speak
            return agents[3]  # "C0"
        elif "NEXT: C1" in messages[-1]["content"]:
            # If the last message has redirect to internal tream member, let member to speak
            return agents[7]  # "C1"
        elif "NEXT: C2" in messages[-1]["content"]:
            # If the last message has redirect to internal tream member, let member to speak
            return agents[8]  # "C2"
        else:
            # if nothing matches, always let P0 to speak
            return agents[0]
    elif last_speaker is agents[7]:  # C1
        if "NEXT: C2" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[8]
        else:
            # All other cases redirect C1 and C2 messages to C0
            return agents[6]
    elif last_speaker is agents[8]:  # C2
        if "NEXT: C1" in messages[-1]["content"]:
            # If the last message has redirect to internal team member, let member to speak
            return agents[7]
        else:
            # All other cases redirect C1 and C2 messages to C0
            return agents[6]
    else:
        # if nothing matches, always let P0 to speak
        return agents[0]
        # return "random"

Now the chat ends correctly. However, sometimes, I get this behaviour (I changed the prompt a little bit but it's not important):

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-12-04 18:57:03,715 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-12-04 18:57:03,861 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel comics, to manga and graphic novels, all play a part in enriching the minds of the young and the young-at-heart.

...


--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

UPDATE CONTEXT

--------------------------------------------------------------------------------

Next speaker: P0

Updating context and resetting conversation.
VectorDB returns doc_ids:  [['45f5db71', '42156ee1', '53c76a5a']]
Adding content of doc 42156ee1 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: 9. Review Panel Descriptions
It will take many drafts of the comic to get it right, and most writers work with their editor until each page is perfect before the artist begins to draw. The lettering is usually left until last so that once the images are in place the writer can tweak the dialogue to fit the frame.


...


--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

UPDATE CONTEXT

--------------------------------------------------------------------------------

Next speaker: P0

Updating context and resetting conversation.
Adding content of doc 53c76a5a to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: SPECTACULARLY VARIED THE SECRET IS NOTIN
PAST 1S ANY INDICATION, WHAT THE DEFINITION
COMICS: AUTURE WILL BE SAYS BUT IN WHAT IT

...


--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

UPDATE CONTEXT

--------------------------------------------------------------------------------

Next speaker: P0

Updating context and resetting conversation.
VectorDB returns doc_ids:  [['45f5db71', '42156ee1', '53c76a5a']]
No more context, will terminate.
P0 (to chat_manager):

TERMINATE

--------------------------------------------------------------------------------
==============================
Chat Ends
==============================

Process finished with exit code 0

It seems that sometimes it doesn't understand context, the context gets updated until it runs out of context and terminates.

Is it a normal behaviour?

thinkall · 2024-12-05T00:15:17Z

Hi @gianx89 , it's normal with the current default prompt of the RAG agent. You can either try adding something like "never reply with 'update context'" to your question ; or pass in a new simple system message to the rag agent (I'd recommend this since your question already have detailed prompts in it). You can find an example in the notebooks and docs I've shared in my previous response.

github-actions bot added the needs-triage label Nov 28, 2024

MohMaz added the proj-agentchat label Nov 28, 2024

MohMaz added this to the 0.4.1 milestone Nov 29, 2024

MohMaz removed the needs-triage label Nov 29, 2024

ekzhu assigned thinkall Dec 2, 2024

ekzhu added 0.2 Issues which are related to the pre 0.4 codebase rag retrieve-augmented generative agents and removed proj-agentchat labels Dec 2, 2024

ekzhu removed this from the 0.4.1 milestone Dec 2, 2024

thinkall mentioned this issue Dec 4, 2024

Add warning message when NoEligibleSpeaker #4535

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors #4422

When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors #4422

gianx89 commented Nov 28, 2024 •

edited

Loading

MohMaz commented Nov 28, 2024

gianx89 commented Nov 29, 2024

ekzhu commented Nov 29, 2024 •

edited

Loading

gianx89 commented Nov 29, 2024

gianx89 commented Nov 29, 2024

thinkall commented Dec 3, 2024

gianx89 commented Dec 3, 2024 •

edited

Loading

thinkall commented Dec 4, 2024 •

edited

Loading

gianx89 commented Dec 4, 2024

thinkall commented Dec 5, 2024

When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors #4422

When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors #4422

Comments

gianx89 commented Nov 28, 2024 • edited Loading

MohMaz commented Nov 28, 2024

gianx89 commented Nov 29, 2024

Reproducing the Problem

ekzhu commented Nov 29, 2024 • edited Loading

gianx89 commented Nov 29, 2024

gianx89 commented Nov 29, 2024

thinkall commented Dec 3, 2024

gianx89 commented Dec 3, 2024 • edited Loading

thinkall commented Dec 4, 2024 • edited Loading

gianx89 commented Dec 4, 2024

thinkall commented Dec 5, 2024

gianx89 commented Nov 28, 2024 •

edited

Loading

ekzhu commented Nov 29, 2024 •

edited

Loading

gianx89 commented Dec 3, 2024 •

edited

Loading

thinkall commented Dec 4, 2024 •

edited

Loading