Customisation of speaker select message and prompt in GroupChat

[marklysze]

Why are these changes needed?

During a Group Chat the Select Speaker Message (first in the context) and Select Speaker Prompt (last in the context) are hardcoded f-strings based on a role-playing prompt. To provide greater flexibility in the final content (for both OpenAI and especially non-OpenAI models), I'm proposing the ability to customise both messages (while including the ability to include the role descriptions and list).

From my testing with open source/weight LLMs I have found that changing these can influence the LLM noticeably.

Changes

I'm proposing the addition of two new Optional[str] (default is None) attributes of the GroupChat class:
custom_select_speaker_msg
custom_select_speaker_prompt

These will be used in the GroupChat select_speaker_msg and select_speaker_prompt functions, which will return the current prompt if the respective attributes are None.

custom_select_speaker_msg can include {roles} in its string, which will be replaced with self._participant_roles(agents), and it can include {agentlist} which will be replaced with {[agent.name for agent in agents]}.

custom_select_speaker_prompt can include {agentlist} which will be replaced with {[agent.name for agent in agents]}.

This enables you to replicate the insertion of role descriptions and role lists as per the current prompts.

I have not added a documentation example but will do if we are pushing forward with this change.

Related issue number

This is related to my Draft PR #1746.

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.58%. Comparing base (7a685b5) to head (1a2f458).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2182       +/-   ##
===========================================
+ Coverage   37.55%   49.58%   +12.02%     
===========================================
  Files          77       77               
  Lines        7719     7730       +11     
  Branches     1657     1795      +138     
===========================================
+ Hits         2899     3833      +934     
+ Misses       4574     3578      -996     
- Partials      246      319       +73     

Flag	Coverage Δ
unittest	14.21% <15.38%> (?)
unittests	48.51% <100.00%> (+10.96%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ekzhu]

Some test errors in CI, could you check the test. It looks like the whitespaces in the expected string doesn't match the output.

>       assert (
            groupchat.select_speaker_msg()
            == """You are the CEO of a niche organisation creating small software tools for the healthcare sector with a small team of specialists. Call them in sequence.
    The job roles and responsibilities are:
    Alice: A wonderful employee named Alice.
    Bob: An amazing employee named Bob.
    
    You must select only from ['Alice', 'Bob']."""
        )
E       assert "You are the ...ice', 'Bob']." == "You are the ...ice', 'Bob']."
E         
E         Skipping [143](https://github.com/microsoft/autogen/actions/runs/8485511517/job/23250435999?pr=2182#step:7:144) identical leading characters in diff, use -v to show
E           sequence.
E         - The job roles and responsibilities are:
E         +     The job roles and responsibilities are:
E         ? ++++
E         - Alice: A wonderful employee named Alice.
E         +     Alice: A wonderful employee named Alice.
E         ? ++++
E           Bob: An amazing employee named Bob.
E           
E         - You must select only from ['Alice', 'Bob'].
E         +     You must select only from ['Alice', 'Bob'].
E         ? ++++

test/agentchat/test_groupchat.py:1222: AssertionError

[marklysze]

Some test errors in CI, could you check the test. It looks like the whitespaces in the expected string doesn't match the output.

>       assert (
            groupchat.select_speaker_msg()
            == """You are the CEO of a niche organisation creating small software tools for the healthcare sector with a small team of specialists. Call them in sequence.
    The job roles and responsibilities are:
    Alice: A wonderful employee named Alice.
    Bob: An amazing employee named Bob.
    
    You must select only from ['Alice', 'Bob']."""
        )
E       assert "You are the ...ice', 'Bob']." == "You are the ...ice', 'Bob']."
E         
E         Skipping [143](https://github.com/microsoft/autogen/actions/runs/8485511517/job/23250435999?pr=2182#step:7:144) identical leading characters in diff, use -v to show
E           sequence.
E         - The job roles and responsibilities are:
E         +     The job roles and responsibilities are:
E         ? ++++
E         - Alice: A wonderful employee named Alice.
E         +     Alice: A wonderful employee named Alice.
E         ? ++++
E           Bob: An amazing employee named Bob.
E           
E         - You must select only from ['Alice', 'Bob'].
E         +     You must select only from ['Alice', 'Bob'].
E         ? ++++

test/agentchat/test_groupchat.py:1222: AssertionError

I'm not sure why that is failing, it should match, let me try running tests again.

[ekzhu]

Check the test failures in CI.

[ekzhu]

We can't run Open AI tests in CI from branches outside of the Microsoft repo.

Can you push your code to a new branch in the Microsoft repo, and create a new PR referencing this one? So we can run the OpenAi test

[marklysze]

We can't run Open AI tests in CI from branches outside of the Microsoft repo.

Can you push your code to a new branch in the Microsoft repo, and create a new PR referencing this one? So we can run the OpenAi test

Thanks, I'm working to do this now.

[marklysze]

I've put the code changes into a branch in this repository (branch: "2182_custom_select_speaker"), and created a new PR, #2233.

Let me know if I can close this now.